Billing System Issues
Incident Report for Hologram
Postmortem

Summary
On 08/02/18 a routine database change caused the Hologram billing system to overbill certain customers for data usage sessions. The change prevented the billing system from properly updating new data usage sessions as billed. When identified, Hologram's Engineering team corrected the database change to restore accurate data usage billing and refunded customer accounts for errant charges. Hologram Customer Success reached out to affected customers for notification. Hologram network connectivity and other platform services were unaffected.

Cause of Failure
On 08/02/18 a routine change was made in the pricing database. Due to a foreign key mismatch, the change caused an error in the billing system anytime a device on a specific pricing plan was being processed for data usage. The error resulted in the usage being deducted from the customer account balance but failing to complete a transaction to mark the usage as billed. Data usage sessions from other devices during the batch processing also errored out and failed to accurately mark the usage as billed resulting in duplicate charges.

Resolution and Recovery
Initial reports of higher than expected billing charges were escalated to the Hologram Data and Engineering Team who confirmed the issue, identified the affected accounts, patched the database, and refunded any duplicate transactions.

Fixes and Next Steps:
Technical debrief with engineering team (Complete 08/02/18)
Outreach to affected accounts (Complete 08/02/18)
Technical scoping for billing exception updates (Complete 08/06/18)
Preventative billing improvements (In Progress - 08/06/18)

Posted 9 months ago. Aug 08, 2018 - 17:50 UTC

Resolved
Billing system restored to normal service level. Affected customers will be contacted with further information.
Posted 9 months ago. Aug 02, 2018 - 16:53 UTC
Monitoring
Issue has been fixed and billing has resumed. Team continuing to monitor.
Posted 9 months ago. Aug 02, 2018 - 16:43 UTC
Update
Engineering has identified the issue and is working on a fix to restore billing. Customer success will reach out to affected customers for any required account reconciliation.
Posted 9 months ago. Aug 02, 2018 - 15:57 UTC
Identified
We're investigating reports of billing system issues that could cause improper data billing for certain customers. Billing will be temporarily paused while the engineering team investigates. Network and SIM operations unaffected.
Posted 9 months ago. Aug 02, 2018 - 15:55 UTC