At approximately 6am EST a global outage occurred on the network of CenturyLink (https://www.centurylink.com/). CenturyLink operates a very large portion of the data circuits that make up the world's global commercial networks. Read more
The root cause of the outage was a bug in our Intrusion Detection and Prevention System. This bug, under a unique and improbable set of circumstances we've identified, resulted in system memory exhaustion on many customer-facing systems. As we have never encountered this issue before, it took approximately 30 minutes to diagnose the problem and about 45 minutes for all systems to return to normal operation once a fix was put in place. Read more
Root cause has been fixed for all sites. Detailed summary of what happend will be provided shortly.
Service restoration complete for a majority of clients. We're working on small remainder of clients impacted.
Service restoration is underway on rolling basis.
Our team has identified the source of the problem and are working on a resolution.
Our Hosting team is investigating a network issue causing site outages.