tag:status.hologram.io,2005:/historyHologram Status - Incident History2024-03-29T00:06:24ZHologramtag:status.hologram.io,2005:Incident/203325042024-03-26T20:10:02Z2024-03-26T20:10:02ZUpcoming Maintenance for SIMs in 8935* Range<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>20:10</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>20:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>21:40</var> UTC</small><br><strong>Scheduled</strong> - Together with one of our partners, we are adding an additional data center in Europe for increased geographical redundancy for SIMs with a profile whose ICCID begins with 8935* (G2).<br /><br />Please be aware that by adding an additional data center in a new geography, the public IP address range associated with your devices may change. If you maintain lists of permitted/allowed public IP ranges you need to add a new range ahead of the planned upgrade on March 26.<br /><br />Notice of the new IPs was sent to account owners and our customer support team can provide the required information.<br /><br />The maintenance will occur on March 26, 2024 at approximately 20:00 UTC. We anticipate the maintenance to last approximately 10 minutes within this window. While we do not expect any disruption in connectivity during the maintenance window, it's important to note that the public IP address associated with your devices may change.<br /><br />There will be additional maintenance events scheduled for:<br /><br />April 2nd.- We will start forwarding data through the new data center in preparation for the maintenance window of April 8th (below)<br /><br />April 8th at approximately 06:00 UTC there will be maintenance work done to the original data center in order to improve the system and services. The duration of this maintenance window will be approximately 7 hours, and there will be two brief disconnections on this data center of 25 minutes each. We don't expect any downtime as our second data center will be up and running handling traffic.</p>tag:status.hologram.io,2005:Incident/203030902024-03-20T04:17:42Z2024-03-20T04:17:42ZDelayed SIM Operations<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>04:17</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>17:29</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>17:28</var> UTC</small><br><strong>Update</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>17:28</var> UTC</small><br><strong>Identified</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p>tag:status.hologram.io,2005:Incident/202709402024-03-16T18:29:47Z2024-03-16T18:29:47ZPartial network outage in West/Central Africa<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>18:29</var> UTC</small><br><strong>Resolved</strong> - Connectivity has been restored in the region</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>00:25</var> UTC</small><br><strong>Monitoring</strong> - Connectivity has been restored for almost all affected devices in the region and we are continuing to monitor.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>16:15</var> UTC</small><br><strong>Investigating</strong> - There's a major internet outage affecting West and Central Africa, which might impact cellular connectivity services in some of these countries.</p>tag:status.hologram.io,2005:Incident/202518672024-03-15T15:30:19Z2024-03-15T15:30:19ZCellular connectivity interruption on some SIM cards<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>15:30</var> UTC</small><br><strong>Resolved</strong> - We've been monitoring the fix from our upstream providers and we can confirm the issue has been resolved. We will provide more information as we receive it and review the root cause analysis.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>15:29</var> UTC</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>05:27</var> UTC</small><br><strong>Monitoring</strong> - Our upstream provider has reported the issue is fixed and we are now monitoring to ensure that connectivity has been completely restored.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>01:08</var> UTC</small><br><strong>Investigating</strong> - We are seeing an issue preventing some SIM cards with the 89445* prefix from being able to attach to the network and pass data.. We have escalated this to upstream carrier partners and they are actively investigating.</p>tag:status.hologram.io,2005:Incident/200503032024-02-23T01:23:09Z2024-02-23T01:23:09ZMajor nationwide cellular network incident in the US<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>01:23</var> UTC</small><br><strong>Resolved</strong> - Services appear to be operational again and we've seen overage connectivity health in the US recover. We'll still monitor any further news and events and support our customers.</p><p><small>Feb <var data-var='date'>22</var>, <var data-var='time'>17:58</var> UTC</small><br><strong>Identified</strong> - There's a major nationwide incident in the US, this incident is directly on local carriers infrastructure mostly affecting AT&T but also some T-Mobile and Verizon users.<br /><br />We're monitoring the situation closely and keeping open communications with our network partners.</p>tag:status.hologram.io,2005:Incident/198702832024-01-31T15:33:00Z2024-01-31T15:33:00ZDelayed SIM Operations<p><small>Jan <var data-var='date'>31</var>, <var data-var='time'>15:33</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>21:17</var> UTC</small><br><strong>Investigating</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers (activations, suspensions, data usage reports, etc). Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p>tag:status.hologram.io,2005:Incident/198695702024-01-30T20:03:54Z2024-01-30T20:03:54ZDelayed SIM Operations<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>20:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>19:29</var> UTC</small><br><strong>Investigating</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers (activations, suspensions, data usage reports, etc). Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p>tag:status.hologram.io,2005:Incident/197704612024-01-20T00:57:51Z2024-01-20T00:57:51ZDelayed SIM Operations<p><small>Jan <var data-var='date'>20</var>, <var data-var='time'>00:57</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>19</var>, <var data-var='time'>22:32</var> UTC</small><br><strong>Identified</strong> - We've identified a backlog on our SIM operations.<br /><br />Activations, state changes and plan changes might be delayed while the system processes the backlog.</p>tag:status.hologram.io,2005:Incident/194432092023-12-15T09:00:14Z2023-12-15T09:00:14ZUpcoming work, short impact on sims range 89357*<p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>09:00</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>15</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>14</var>, <var data-var='time'>22:47</var> UTC</small><br><strong>Scheduled</strong> - We will work to improve the stability, security and performance of our infrastructure.<br /><br />Sims with the 89357* prefix active data sessions might be impacted during the service restart (up to 5 seconds).</p>tag:status.hologram.io,2005:Incident/193971372023-12-11T16:01:56Z2023-12-11T16:01:57ZDelayed usage data for some SIMs<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:01</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>05:48</var> UTC</small><br><strong>Monitoring</strong> - We are seeing usage data recovering and are continuing to monitor</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>04:32</var> UTC</small><br><strong>Identified</strong> - The carrier partner has confirmed the issue and is working on a fix</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>03:26</var> UTC</small><br><strong>Investigating</strong> - We have detected an issue with an upstream carrier partner causing some usage data to be delayed.<br />There does not seem to be an issue with any cellular connectivity at this time and the issue is only with usage reporting and billing. We have escalated to our partners and are monitoring the situation.</p>tag:status.hologram.io,2005:Incident/193168192023-12-04T18:03:12Z2023-12-04T18:03:12ZDelayed SIM Operations<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>18:03</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>17:14</var> UTC</small><br><strong>Investigating</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p>tag:status.hologram.io,2005:Incident/191964732023-11-22T18:29:58Z2023-11-22T18:29:58ZEmergency Maintenance Update for ICCID Range 89357*<p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>18:29</var> UTC</small><br><strong>Completed</strong> - Everything is operational and there was no major impact.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>18:07</var> UTC</small><br><strong>Verifying</strong> - The maintenance update has been completed, no major impact was observed, we will monitor and then proceed to mark it as closed.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>17:39</var> UTC</small><br><strong>In progress</strong> - There was a timezone mistake on the notification provided to us and the Maintenance Update just started, it was one hour earlier than the original expected time.</p><p><small>Nov <var data-var='date'>22</var>, <var data-var='time'>17:34</var> UTC</small><br><strong>Scheduled</strong> - An upstream partner has notified us they'll be doing an emergency update in one of our platforms , and they advised there might be a short outage during their work, dropping current GTP sessions.<br /><br />Services won't be impacted for the entire maintenance duration, we're expecting up to a 10 minute duration but should be shorter.<br /><br />We'll be providing updates as we have them.</p>tag:status.hologram.io,2005:Incident/191382022023-11-21T08:31:35Z2023-11-21T08:31:35ZScheduled maintenance for sims in the 89357* range<p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>08:31</var> UTC</small><br><strong>Completed</strong> - Everything seems to be up and running properly, we will mark this Maintenance Window as completed.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>08:23</var> UTC</small><br><strong>Verifying</strong> - The planned work has finished and we are confirming a few things on our end. The impact was minimum and only lasted for 2 to 3 minutes.</p><p><small>Nov <var data-var='date'>21</var>, <var data-var='time'>08:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>14:54</var> UTC</small><br><strong>Scheduled</strong> - The objective of this maintenance window is to improve the service of our network. This will expand the geo-redundancy solution for our sims with the ICCID range of 89357*.<br /><br />This will impact existing and new data sessions, with a disruption of up to 30 minutes.<br /><br />We apologize for any inconvenience this may cause, we are working to enhance the reliability of the network.</p>tag:status.hologram.io,2005:Incident/190258172023-11-05T19:04:34Z2023-11-05T19:04:34ZNetwork and connectivity incident on SIMs on the 89357* range<p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>19:04</var> UTC</small><br><strong>Resolved</strong> - We've been monitoring the fix and we can confirm the issue has been resolved. We will provide more information when we get and review the root cause analysis.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:49</var> UTC</small><br><strong>Monitoring</strong> - The issue has been fixed, we're still monitoring that all configurations are properly done and that there's no remaining issues. We will also work with responsible parties to get a root cause analysis .</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>17:39</var> UTC</small><br><strong>Update</strong> - We've been advised that our partners provider is still working on resolving the issue, they're past the ETA provided. We're all working on resolving this as soon as possible.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>15:42</var> UTC</small><br><strong>Identified</strong> - The issue was identified on a backbone link of one of our partners providers.<br />We are constantly pushing to obtain more details regarding this incident and will provide updates as we have more information.</p><p><small>Nov <var data-var='date'> 5</var>, <var data-var='time'>11:14</var> UTC</small><br><strong>Investigating</strong> - We are seeing an issue with a network partner causing some SIM cards on the 89357* ICCID range to be unable to pass data on the network. We immediately reported to our upstream partner and received confirmation about an incident on their side, we're all investigating and will provide more details about impact and path to resolution.</p>tag:status.hologram.io,2005:Incident/185474122023-09-19T22:01:08Z2023-09-19T22:01:08ZCellular connectivity interruption on some SIM cards<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>22:01</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>21:11</var> UTC</small><br><strong>Monitoring</strong> - We are seeing traffic properly being rerouted away from the broken link and succeeding, and close to normal levels. We will continue monitoring this new route with our partners.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>20:03</var> UTC</small><br><strong>Update</strong> - After reviewing different approaches with our upstream partners, they have started routing traffic via a different link. We don't have an ETA yet to full resolution but we are seeing some 4G registrations succeed.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>17:29</var> UTC</small><br><strong>Identified</strong> - Our upstream partners have traced the incident to a cut MedNautilus fiber cable and are working to reroute traffic</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>16:36</var> UTC</small><br><strong>Investigating</strong> - We are seeing an issue preventing some SIM cards with the 8935* prefix from being able to attach to the network and pass data. We have escalated this to upstream carrier partners and they are actively investigating.</p>tag:status.hologram.io,2005:Incident/182975632023-09-06T16:40:43Z2023-10-03T15:03:34ZConnectivity Interruption on some SIM cards<p><small>Sep <var data-var='date'> 6</var>, <var data-var='time'>16:40</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. Working with our upstream partner we have identified and addressed the root causes. Device connections have returned to normal levels and we are seeing consistent performance across our network. A very small number of devices could still be experiencing unusual behavior. If you have such a device, please contact support for device-specific diagnostics.</p><p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>17:14</var> UTC</small><br><strong>Update</strong> - Our data shows that 99% of devices are connecting normally now after the weekend.<br /><br />We're still troubleshooting a small subset of devices that have unstable data connections (sessions dropping more often than expected).</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>16:56</var> UTC</small><br><strong>Update</strong> - Our data shows that 97% of devices are connecting normally. We have observed a small subset of devices with unstable data connections, with sessions dropping more often than expected. We are diagnosing root causes.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>00:23</var> UTC</small><br><strong>Update</strong> - Over 95% of affected devices have restored connectivity, with more devices coming back online each hour. We're still working with our upstream partners to ensure the same success on remaining devices.<br />We have observed a small subset of devices with unstable data connections, with sessions dropping more often than expected. We are diagnosing root causes</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>21:11</var> UTC</small><br><strong>Update</strong> - Based on our monitoring we've identified 90% of our affected SIM cards connectivity has been restored. We're still working with our upstream partners to ensure the same success on the remaining ones.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>19:38</var> UTC</small><br><strong>Update</strong> - Based on our monitoring we've identified 85% of our affected SIM cards connectivity has been restored. We're still working with our upstream partners to ensure the same success on the remaining ones.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>13:09</var> UTC</small><br><strong>Update</strong> - Our upstream partner is continuing to make improvements, and Hologram is starting to see devices come on line. To reduce signal overload on the network, the number of new devices are being rate limited to prevent congestion, and our upstream partner is continuing to incrementally increase capacity. Both Hologram and our partner are closely monitoring the network recovery to ensure network stability, and work through the recovery in a controlled manner.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>09:28</var> UTC</small><br><strong>Monitoring</strong> - Our upstream partner is continuing to make improvements, and Hologram is starting to see devices come on line. To reduce signal overload on the network, the number of new devices are being rate limited to prevent congestion, and our upstream partner is continuing to incrementally increase capacity. Both Hologram and our partner are closely monitoring the network recovery to ensure network stability, and work through the recovery in a controlled manner.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>07:42</var> UTC</small><br><strong>Update</strong> - Our upstream partner is continuing to implement changes to bring devices back online, and is currently working to mitigate signaling congestion. With the current congestion, Hologram continues to see devices failing to register with carrier networks. We are continuing to monitor the situation.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>05:19</var> UTC</small><br><strong>Update</strong> - Our upstream partner has shifted traffic to a new node and is beginning to see improvements in IoT device connectivity. We are continuing to monitor the situation. We will continue to post updates here as we learn more.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>02:01</var> UTC</small><br><strong>Identified</strong> - The previously identified fix for the failures in our upstream partner's network interfaces was not ultimately viable and was not implemented. Our partner believes they have now identified a common problem with the interface failures and are working on a fix.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>01:06</var> UTC</small><br><strong>Update</strong> - The likely root cause of the regression has been identified on a node in our upstream partner's network interface. They engaged with their vendors and a solution has been identified, which will be executed in the next 30 minutes. The updated node will come back online and traffic will be gradually increased to it to ensure a stable recovery.<br />The expected recovery time depends on the depth of backlog of connection requests, it will be a slow release to avoid overloading the signal, they're estimating recovery by 0700 to 0800 UTC.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>22:59</var> UTC</small><br><strong>Update</strong> - Our upstream partner has identified the root cause of the regression. One of the provider's network interface was unable to support the amount of traffic being released and was compromised.<br />The solution is currently being assessed, in which they'll migrate the signaling services to a different node that is operating normally and has bandwidth to support the incremental traffic.<br />Latest estimate for full resolution is 0700 to 0800 UTC. This estimate is based on the assumption that the applied solutions work as intended.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>20:59</var> UTC</small><br><strong>Update</strong> - After trending upwards close to a resolution, we noticed a regression on resolved sims. We escalated with our upstream partners and they confirmed there has been a major regression on the solution that was implemented. We're monitoring the impact and working with our partners to identify the root cause of this regression.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>19:01</var> UTC</small><br><strong>Update</strong> - We've been monitoring progress and it's been steadily improving. We're already past 65% of resolution , we will keep monitoring and updating.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>15:58</var> UTC</small><br><strong>Monitoring</strong> - The latest update received from our upstream partner indicates signalling has been stabilized and congestion is now overcome with continuous monitoring ongoing. The network is processing traffic as per normal standard, though affected devices will take some time to all reattach.<br />From timelines of previous similar incidents, it is expected that full recovery will likely take until well past midnight UTC. <br />All teams are focusing on improving these timelines wherever possible.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>12:39</var> UTC</small><br><strong>Update</strong> - There are still issues with the connectivity on our upstream partners network.<br /><br />The cause of this incident has now been rectified and stability has been confirmed.<br /><br />They're now facing a signaling storm due to congestion, they're restricting the traffic and are slowly increasing throughput to resolve this.<br /><br />We're still unable to provide an ETA but we started seeing improvements.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>09:23</var> UTC</small><br><strong>Update</strong> - Our upstream partner continues to work on the remaining network instability issues adversely affecting subscriber attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>08:11</var> UTC</small><br><strong>Update</strong> - Our upstream partner continues to work on the remaining network instability issues adversely affecting subscriber attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>07:26</var> UTC</small><br><strong>Update</strong> - Our upstream partners have stabilized the replacement hardware and continue to work on the remaining network instability issues that are adversely affecting subscribe attachments. There is no ETA yet on resolution. We will continue to post updates here as we learn more.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>07:25</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>06:58</var> UTC</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>05:39</var> UTC</small><br><strong>Update</strong> - Our upstream partners have replaced faulty hardware, have begun bringing interconnect links back online, and are continuing to work to resolve the issues that remain. Unfortunately, bringing back the interconnect links have not yet had the expected effect on connectivity. Subscriber attachments are still adversely affected. There is no ETA yet on resolution. We will continue to post updates here as we learn more.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>04:31</var> UTC</small><br><strong>Update</strong> - Our upstream partners have replaced faulty hardware, have begun bringing interconnect links back online, and are continuing to work to resolve the issues that remain. Subscriber attachments are still adversely affected. We will continue to post updates here as we learn more.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>03:25</var> UTC</small><br><strong>Update</strong> - Our upstream partners have replaced faulty hardware and are continuing to work to resolve the issues that remain. We will continue to post updates here as we learn more.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>02:35</var> UTC</small><br><strong>Update</strong> - Unfortunately the fix attempted by our upstream partners did not resolve the issue. They are continuing to investigate. We will post updates here as we learn more.</p><p><small>Aug <var data-var='date'>29</var>, <var data-var='time'>00:55</var> UTC</small><br><strong>Update</strong> - Upstream partners have identified the issue and are implementing a fix.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>23:45</var> UTC</small><br><strong>Identified</strong> - The issue has been identified. We will post updates here as we learn more.</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>19:47</var> UTC</small><br><strong>Update</strong> - This issue is affecting SIMs starting with the 8944* prefix. It appears to be preventing any attachment to the cellular network right now. This has been escalated to the highest level</p><p><small>Aug <var data-var='date'>28</var>, <var data-var='time'>19:39</var> UTC</small><br><strong>Investigating</strong> - We are seeing an issue with a network partner causing some SIM cards to be unable to pass data on the network. The issue has been escalated and we are investigating along with our partner. We will post updates here as we learn more.</p>tag:status.hologram.io,2005:Incident/183554522023-09-05T16:26:24Z2023-09-05T16:26:24ZNetwork and connectivity interruption for some SIMs on the 8944* range<p><small>Sep <var data-var='date'> 5</var>, <var data-var='time'>16:26</var> UTC</small><br><strong>Resolved</strong> - We can confirm the incident has been resolved.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>14:48</var> UTC</small><br><strong>Monitoring</strong> - The issue was identified and fixed by the upstream partner. We are seeing steady improvements with devices coming back online every hour. We will continue to monitor.</p><p><small>Sep <var data-var='date'> 2</var>, <var data-var='time'>00:49</var> UTC</small><br><strong>Investigating</strong> - We are seeing an issue with a network partner causing some SIM cards on the 8944* ICCID range to be unable to pass data on the network. The issue has been escalated and we are investigating along with our partner. We will post updates here as we learn more.</p>tag:status.hologram.io,2005:Incident/183405132023-08-31T22:59:58Z2023-08-31T22:59:58ZNetwork and connectivity incident on SIMs on the 89357* range<p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>22:59</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>22:59</var> UTC</small><br><strong>Update</strong> - We've determined the incident has been resolved and connectivity on the sim range 89357* has been restored.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>22:45</var> UTC</small><br><strong>Monitoring</strong> - The issue was promptly identified and fixed, we see almost full recovery, we will keep monitoring.</p><p><small>Aug <var data-var='date'>31</var>, <var data-var='time'>21:46</var> UTC</small><br><strong>Investigating</strong> - We are seeing an issue with a network partner causing some SIM cards on the 89357* ICCID range to be unable to pass data on the network. We immediately reported to our upstream partner and received confirmation about an incident on their side, we're all investigating and will provide more details about impact and path to resolution.</p>tag:status.hologram.io,2005:Incident/182480352023-08-27T11:46:10Z2023-08-27T11:46:10ZPlanned Maintenance Window with Brief Data Service Interruption<p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>11:46</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>11:40</var> UTC</small><br><strong>Verifying</strong> - We are doing some final checks but the network appears to have returned to normal operation</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>11:04</var> UTC</small><br><strong>Update</strong> - We are seeing data sessions recover and are monitoring with our partner to confirm maintenance has completed successfully</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>09:05</var> UTC</small><br><strong>Update</strong> - Signaling outage completed as planned and devices are coming back online. Data outage will be happening shortly</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>07:19</var> UTC</small><br><strong>Update</strong> - Our partner has informed us the downtime will be about an hour later than expected so initial brief signaling downtime at around 0800 UTC and data downtime at around 1000 UTC</p><p><small>Aug <var data-var='date'>27</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>15:34</var> UTC</small><br><strong>Update</strong> - Our partner has given us more details about the maintenance period so we can give more accurate estimates of service interruptions.<br /><br />They expect that near the beginning of the maintenance window (0700 UTC) there will be an interruption in signaling traffic that will prevent new tower attachments for several minutes. Any existing sessions will not be interrupted at that time.<br /><br />At around 2 hours into the window (approximately 0900 UTC) there will be an interruption in data services that will last anywhere from a few minutes to a maximum of one hour depending on caching settings with the local carrier.</p><p><small>Aug <var data-var='date'>24</var>, <var data-var='time'>16:20</var> UTC</small><br><strong>Scheduled</strong> - An upstream partner will be performing maintenance on their network which will cause an interruption in cellular data service. This will affect SIMs with the ICCID prefix 8935*<br /><br />Maintenance will be occurring during this entire window but expected downtime will only be for a few minutes during the window. SIMs that are affected will experience a brief interruption in open data sessions and tower registrations but should reattach and resume data sessions very quickly.<br /><br />We will be monitoring the progress of the maintenance with our partner and provide updates as appropriate</p>tag:status.hologram.io,2005:Incident/182637242023-08-25T20:45:44Z2023-08-25T20:45:44ZDelayed SIM Operations<p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>20:45</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>25</var>, <var data-var='time'>19:02</var> UTC</small><br><strong>Investigating</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p>tag:status.hologram.io,2005:Incident/181971452023-08-20T16:30:10Z2023-08-20T16:30:10ZCellular Connectivity Issue With Certain SIMs<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>16:30</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>19</var>, <var data-var='time'>22:17</var> UTC</small><br><strong>Investigating</strong> - Our network monitoring systems have detected an issue with an upstream provider that is causing some SIMs that start with the 8935 ICCID prefix to have issues attaching to the network. Not all SIMs in this range are affected, but some subset is experiencing a problem. This has been escalated with the provider and they are actively working on the issue.</p>tag:status.hologram.io,2005:Incident/180965552023-08-11T15:22:22Z2023-08-11T15:22:22ZDelayed SIM Operations<p><small>Aug <var data-var='date'>11</var>, <var data-var='time'>15:22</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>16:49</var> UTC</small><br><strong>Monitoring</strong> - A fix has been identified and is being implemented by our upstream partner now. We will be monitoring to ensure any operations issued during the incident window are completed.</p><p><small>Aug <var data-var='date'> 9</var>, <var data-var='time'>15:36</var> UTC</small><br><strong>Identified</strong> - One of our upstream partners is experiencing an issue that is causing some device operations (such as pauses and activations) as well as SMS to experience significant delays. Cellular connectivity for devices remains unaffected. The issue has been escalated with our partner.</p>tag:status.hologram.io,2005:Incident/180134352023-08-01T01:30:26Z2023-08-01T01:30:26ZDelayed SIM Operations<p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>01:30</var> UTC</small><br><strong>Resolved</strong> - All SIMs that were in the queue have been activated and systems are operating normally</p><p><small>Aug <var data-var='date'> 1</var>, <var data-var='time'>01:25</var> UTC</small><br><strong>Monitoring</strong> - Our upstream partner has implemented a fix and the queue is catching up. We are monitoring the remaining activations to make sure everything is processing correctly.</p><p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>22:08</var> UTC</small><br><strong>Update</strong> - We have determined that this issue is being caused by a failure at an upstream partner and have determined that this could cause connectivity issues for some SIMs that are transitioning from Preflight to fully activated. The issue has been escalated with our partner and they are investigating.<br />Cellular data is currently degraded for SIMs starting with 8944 that are in a preflight state.</p><p><small>Jul <var data-var='date'>31</var>, <var data-var='time'>19:21</var> UTC</small><br><strong>Investigating</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p>tag:status.hologram.io,2005:Incident/179176962023-07-21T01:51:05Z2023-07-21T01:51:05ZDelayed SIM Operations<p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>01:51</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>21</var>, <var data-var='time'>01:20</var> UTC</small><br><strong>Monitoring</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p>tag:status.hologram.io,2005:Incident/179169702023-07-20T23:40:54Z2023-07-20T23:40:54ZDelayed SIM Operations<p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>23:40</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>20</var>, <var data-var='time'>23:18</var> UTC</small><br><strong>Monitoring</strong> - There is a currently a backlog of operations delaying SIM operations on some carriers. Backlog is processing and some customers may experience delays. Cellular network and other API services are unaffected.</p>