On 2018/11/20 02:30 UTC one of our global packet gateway providers experienced an unplanned outage during routine maintenance. This prevented SIMs from transmitting packet switched data during the incident. Existing active cellular data sessions were unaffected, as only new connections were utilizing the new router. The outage was resolved by the upstream provider and service was restored to full operation as of 03:10 UTC.
At 2018/11/20 02:15 UTC the gateway provider commenced regular maintenance, which involves an offload migration of new data sessions' packet switched (data) traffic from one router to a redundant router at a different site in preparation for installing system updates. Under normal maintenance or load balancing conditions, this occurs seamlessly, without affecting existing or new connections. During this upgrade, a routing failure adjacent to the new delegate router prevented inactive devices from establishing new data sessions successfully. Existing data sessions were unaffected, as their sessions were not being served by the new delegate router.
Automated monitoring systems at Hologram and at the upstream provider both reported failures occurring for newly-created cellular data sessions. These reports of service disruption were escalated to Hologram's engineers, CTO, and Product Lead. Upstream providers were notified and resolution efforts were coordinated, and Hologram Status Page was updated. Upstream provider found a routing configuration issue preventing normal operation at the new delegate router and fixed the issue to restore routing for new data sessions as of 03:10 UTC. After internal monitoring systems and tests were confirmed as clear, Hologram team then confirmed normal system performance before clearing incident status as Resolved.