On 2019/04/23 00:20 UTC Hologram internal network monitors detected an unplanned outage experienced by one of our global packet gateway providers while they were engaged in routine maintenance. As a result, signalling traffic failed for Hologram and other providers worldwide, and affected SIMs were unable to attach to towers and open new sessions at the time. The outage was resolved by the upstream provider and service was restored to full operation as of 02:00 UTC.
At 2019/04/23 00:00 UTC the gateway provider commenced regular maintenance, which involved an offload migration of signalling traffic. Under normal maintenance or fault conditions, this occurs seamlessly, without affecting existing or new connections. During this upgrade, changes being made to offline signalling services erroneously also affected their online counterparts. As a result, HLR/HSS services became unusable and signalling requests failed.
Automated monitoring systems at Hologram and at the upstream provider both reported failures occurring for signalling requests. These reports of service disruption were escalated to Hologram's engineers, CTO, and Product Lead. Upstream providers were notified and resolution efforts were coordinated, and Hologram Status Page was updated. Upstream provider identified and fixed the issue to restore signalling traffic as of 02:00 UTC. Affected devices, including Hologram automated monitoring devices, automatically reconnected successfully following resolution. After internal monitoring systems and tests were confirmed as clear, Hologram team verified normal system performance before clearing incident status as Resolved. Upstream provider has since coordinated with equipment vendor(s) and has successfully tested a revised upgrade plan to ensure isolation in the event of any possible upgrade interruption, and has communicated changes to Hologram.