Cellular Data and Connectivity Outage
Incident Report for Hologram
Postmortem

Summary

On 2018/11/20 02:30 UTC one of our global packet gateway providers experienced an unplanned outage during routine maintenance. This prevented SIMs from transmitting packet switched data during the incident. Existing active cellular data sessions were unaffected, as only new connections were utilizing the new router. The outage was resolved by the upstream provider and service was restored to full operation as of 03:10 UTC.

Cause of Failure

At 2018/11/20 02:15 UTC the gateway provider commenced regular maintenance, which involves an offload migration of new data sessions' packet switched (data) traffic from one router to a redundant router at a different site in preparation for installing system updates. Under normal maintenance or load balancing conditions, this occurs seamlessly, without affecting existing or new connections. During this upgrade, a routing failure adjacent to the new delegate router prevented inactive devices from establishing new data sessions successfully. Existing data sessions were unaffected, as their sessions were not being served by the new delegate router.

Resolution and Recovery

Automated monitoring systems at Hologram and at the upstream provider both reported failures occurring for newly-created cellular data sessions. These reports of service disruption were escalated to Hologram's engineers, CTO, and Product Lead. Upstream providers were notified and resolution efforts were coordinated, and Hologram Status Page was updated. Upstream provider found a routing configuration issue preventing normal operation at the new delegate router and fixed the issue to restore routing for new data sessions as of 03:10 UTC. After internal monitoring systems and tests were confirmed as clear, Hologram team then confirmed normal system performance before clearing incident status as Resolved.

Posted 9 months ago. Dec 03, 2018 - 16:30 UTC

Resolved
The issue has been identified and is resolved. Hologram team is continuing to monitor connectivity and quality of service.
Posted 9 months ago. Nov 20, 2018 - 03:30 UTC
Investigating
We are currently experiencing an outage impacting all cellular data and connectivity. Devices are connecting to towers, but data services are interrupted. SIM operations are currently unaffected. The team is actively investigating.
Posted 9 months ago. Nov 20, 2018 - 03:10 UTC
This incident affected: Cellular Network.