On March 14th, Hologram experienced degraded service, eventually resulting in a loss of connectivity, with one of our connectivity partners. This incident impacted some customers who use SIMs/profiles with ICCIDs beginning with the 8944* prefix. During the incident, devices using these SIMs/profiles experienced a cellular data outage and degraded cellular data connectivity.
On March 14th around 23:00 UTC, one of our connectivity partners encountered degraded signalling service via one of their upstream interconnect providers. This degraded signalling eventually resulted in a disruption of all signalling via that interconnect provider.
The loss of control plane signalling traffic resulted in the inability for affected devices to attach to towers, establish new data connections, and maintain existing data connections. The resulting degraded service and outage affected all SIMs/profiles with ICCIDs beginning with the 8944* prefix that attempted connections during the incident. Customers using other profiles were unaffected.
The upstream interconnect provider restored signalling services by 02:50 UTC on March 15th, at which time affected devices began reconnecting. Hologram monitored progress until all devices were restored.
Root cause analysis is underway, and the interconnect provider and our connectivity partner are working to reproduce the issue in a sandboxed environment. This work is to confirm root cause and determine why performance degradation occurred and did not remain more isolated.
Root cause analysis is ongoing and we will post an update in the coming weeks including the timeline of events.
This incident had a widespread impact, and we take reliability very seriously. While Hologram and its partners implement systems and safeguards to maximize reliability, and we observed an improved recovery during this incident due to prior implemented changes, we will identify any additional areas of improvement and work to uncover any other containment gaps to prevent a recurrence of this type of root cause.
Root cause analysis is ongoing and we will post an update in the coming weeks as to our findings and our next steps for improved containment and prevention based upon what is learned from this incident.
Our aim at Hologram is to provide the most reliable cellular connectivity product available, and we clearly fell short of our customer expectations with this incident. Although the root cause was with an upstream partner, we are ultimately accountable and are deeply sorry for the disruption to our customers who were unable to use cellular data on this profile as expected due to the outage.