Summary
On 08/02/18 a routine database change caused the Hologram billing system to overbill certain customers for data usage sessions. The change prevented the billing system from properly updating new data usage sessions as billed. When identified, Hologram's Engineering team corrected the database change to restore accurate data usage billing and refunded customer accounts for errant charges. Hologram Customer Success reached out to affected customers for notification. Hologram network connectivity and other platform services were unaffected.

Cause of Failure
On 08/02/18 a routine change was made in the pricing database. Due to a foreign key mismatch, the change caused an error in the billing system anytime a device on a specific pricing plan was being processed for data usage. The error resulted in the usage being deducted from the customer account balance but failing to complete a transaction to mark the usage as billed. Data usage sessions from other devices during the batch processing also errored out and failed to accurately mark the usage as billed resulting in duplicate charges.

Resolution and Recovery
Initial reports of higher than expected billing charges were escalated to the Hologram Data and Engineering Team who confirmed the issue, identified the affected accounts, patched the database, and refunded any duplicate transactions.

Fixes and Next Steps:
Technical debrief with engineering team (Complete 08/02/18)
Outreach to affected accounts (Complete 08/02/18)
Technical scoping for billing exception updates (Complete 08/06/18)
Preventative billing improvements (In Progress - 08/06/18)