This incident has been resolved. Working with our upstream partner we have identified and addressed the root causes. Device connections have returned to normal levels and we are seeing consistent performance across our network. A very small number of devices could still be experiencing unusual behavior. If you have such a device, please contact support for device-specific diagnostics.
Posted Sep 06, 2023 - 16:40 UTC
Update
Our data shows that 99% of devices are connecting normally now after the weekend.
We're still troubleshooting a small subset of devices that have unstable data connections (sessions dropping more often than expected).
Posted Sep 05, 2023 - 17:14 UTC
Update
Our data shows that 97% of devices are connecting normally. We have observed a small subset of devices with unstable data connections, with sessions dropping more often than expected. We are diagnosing root causes.
Posted Aug 31, 2023 - 16:56 UTC
Update
Over 95% of affected devices have restored connectivity, with more devices coming back online each hour. We're still working with our upstream partners to ensure the same success on remaining devices. We have observed a small subset of devices with unstable data connections, with sessions dropping more often than expected. We are diagnosing root causes
Posted Aug 31, 2023 - 00:23 UTC
Update
Based on our monitoring we've identified 90% of our affected SIM cards connectivity has been restored. We're still working with our upstream partners to ensure the same success on the remaining ones.
Posted Aug 30, 2023 - 21:11 UTC
Update
Based on our monitoring we've identified 85% of our affected SIM cards connectivity has been restored. We're still working with our upstream partners to ensure the same success on the remaining ones.
Posted Aug 30, 2023 - 19:38 UTC
Update
Our upstream partner is continuing to make improvements, and Hologram is starting to see devices come on line. To reduce signal overload on the network, the number of new devices are being rate limited to prevent congestion, and our upstream partner is continuing to incrementally increase capacity. Both Hologram and our partner are closely monitoring the network recovery to ensure network stability, and work through the recovery in a controlled manner.
Posted Aug 30, 2023 - 13:09 UTC
Monitoring
Our upstream partner is continuing to make improvements, and Hologram is starting to see devices come on line. To reduce signal overload on the network, the number of new devices are being rate limited to prevent congestion, and our upstream partner is continuing to incrementally increase capacity. Both Hologram and our partner are closely monitoring the network recovery to ensure network stability, and work through the recovery in a controlled manner.
Posted Aug 30, 2023 - 09:28 UTC
Update
Our upstream partner is continuing to implement changes to bring devices back online, and is currently working to mitigate signaling congestion. With the current congestion, Hologram continues to see devices failing to register with carrier networks. We are continuing to monitor the situation.
Posted Aug 30, 2023 - 07:42 UTC
Update
Our upstream partner has shifted traffic to a new node and is beginning to see improvements in IoT device connectivity. We are continuing to monitor the situation. We will continue to post updates here as we learn more.
Posted Aug 30, 2023 - 05:19 UTC
Identified
The previously identified fix for the failures in our upstream partner's network interfaces was not ultimately viable and was not implemented. Our partner believes they have now identified a common problem with the interface failures and are working on a fix.
Posted Aug 30, 2023 - 02:01 UTC
Update
The likely root cause of the regression has been identified on a node in our upstream partner's network interface. They engaged with their vendors and a solution has been identified, which will be executed in the next 30 minutes. The updated node will come back online and traffic will be gradually increased to it to ensure a stable recovery. The expected recovery time depends on the depth of backlog of connection requests, it will be a slow release to avoid overloading the signal, they're estimating recovery by 0700 to 0800 UTC.
Posted Aug 30, 2023 - 01:06 UTC
Update
Our upstream partner has identified the root cause of the regression. One of the provider's network interface was unable to support the amount of traffic being released and was compromised. The solution is currently being assessed, in which they'll migrate the signaling services to a different node that is operating normally and has bandwidth to support the incremental traffic. Latest estimate for full resolution is 0700 to 0800 UTC. This estimate is based on the assumption that the applied solutions work as intended.
Posted Aug 29, 2023 - 22:59 UTC
Update
After trending upwards close to a resolution, we noticed a regression on resolved sims. We escalated with our upstream partners and they confirmed there has been a major regression on the solution that was implemented. We're monitoring the impact and working with our partners to identify the root cause of this regression.
Posted Aug 29, 2023 - 20:59 UTC
Update
We've been monitoring progress and it's been steadily improving. We're already past 65% of resolution , we will keep monitoring and updating.
Posted Aug 29, 2023 - 19:01 UTC
Monitoring
The latest update received from our upstream partner indicates signalling has been stabilized and congestion is now overcome with continuous monitoring ongoing. The network is processing traffic as per normal standard, though affected devices will take some time to all reattach. From timelines of previous similar incidents, it is expected that full recovery will likely take until well past midnight UTC. All teams are focusing on improving these timelines wherever possible.
Posted Aug 29, 2023 - 15:58 UTC
Update
There are still issues with the connectivity on our upstream partners network.
The cause of this incident has now been rectified and stability has been confirmed.
They're now facing a signaling storm due to congestion, they're restricting the traffic and are slowly increasing throughput to resolve this.
We're still unable to provide an ETA but we started seeing improvements.
Posted Aug 29, 2023 - 12:39 UTC
Update
Our upstream partner continues to work on the remaining network instability issues adversely affecting subscriber attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.
Posted Aug 29, 2023 - 09:23 UTC
Update
Our upstream partner continues to work on the remaining network instability issues adversely affecting subscriber attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.
Posted Aug 29, 2023 - 08:11 UTC
Update
Our upstream partners have stabilized the replacement hardware and continue to work on the remaining network instability issues that are adversely affecting subscribe attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.
Posted Aug 29, 2023 - 07:26 UTC
Update
We are continuing to work on a fix for this issue.
Posted Aug 29, 2023 - 07:25 UTC
Update
We are continuing to work on a fix for this issue.
Posted Aug 29, 2023 - 06:58 UTC
Update
Our upstream partners have replaced faulty hardware, have begun bringing interconnect links back online, and are continuing to work to resolve the issues that remain. Unfortunately, bringing back the interconnect links have not yet had the expected effect on connectivity. Subscriber attachments are still adversely affected. There is no ETA yet on resolution. We will continue to post updates here as we learn more.
Posted Aug 29, 2023 - 05:39 UTC
Update
Our upstream partners have replaced faulty hardware, have begun bringing interconnect links back online, and are continuing to work to resolve the issues that remain. Subscriber attachments are still adversely affected. We will continue to post updates here as we learn more.
Posted Aug 29, 2023 - 04:31 UTC
Update
Our upstream partners have replaced faulty hardware and are continuing to work to resolve the issues that remain. We will continue to post updates here as we learn more.
Posted Aug 29, 2023 - 03:25 UTC
Update
Unfortunately the fix attempted by our upstream partners did not resolve the issue. They are continuing to investigate. We will post updates here as we learn more.
Posted Aug 29, 2023 - 02:35 UTC
Update
Upstream partners have identified the issue and are implementing a fix.
Posted Aug 29, 2023 - 00:55 UTC
Identified
The issue has been identified. We will post updates here as we learn more.
Posted Aug 28, 2023 - 23:45 UTC
Update
This issue is affecting SIMs starting with the 8944* prefix. It appears to be preventing any attachment to the cellular network right now. This has been escalated to the highest level
Posted Aug 28, 2023 - 19:47 UTC
Investigating
We are seeing an issue with a network partner causing some SIM cards to be unable to pass data on the network. The issue has been escalated and we are investigating along with our partner. We will post updates here as we learn more.
Posted Aug 28, 2023 - 19:39 UTC
This incident affected: Cellular Networking (Global Cellular Data Network).