- Cloudflare admits it caused its own outage – it wasn’t a cyber attack
- Fluctuating bug reports made the problem challenging to identify at first
- The “unacceptable” interruption led to some learning opportunities
Cloudflare has shared more details about its outage on November 18 – its worst outage since 2019 – and confirms that it was not the result of an attack or any other type of malicious activity.
In a blog post, company co-founder and CEO Matthew Prince explained that a database permission change triggered the system to generate a ‘feature file’ that doubled in size before propagating to all the machines on its network, causing the software to fail.
Because Cloudflare was able to identify what had gone wrong, normal operations resumed just over three hours after the outage, with full recovery a few hours later.
Cloudflare confirms that the outage was not an attack
“Core traffic was flowing pretty much as normal by 2:30 p.m.,” Prince wrote, as confirmed by a chat that showed a big drop in 5xx error HTTP status codes right around that time.
However, Cloudflare had to dig a little deeper to find out what exactly was going on due to a fairly high fluctuating range of error reports. This was because the problematic file was generated every five minutes.
“In addition to returning HTTP 5xx errors, we observed significant increases in response latency from our CDN during the impact period,” Prince added, noting that “large amounts of CPU” were being used across debugging and observability.
Cloudflare’s status page also went down during the attack – a page completely independent of Cloudflare’s infrastructure. Apparently this was little more than a coincidence.
Nevertheless, the outage at least served as a learning opportunity for Cloudflare, which now promises to activate more global kill switches for features.
“An outage that today is unacceptable,” Prince concluded, before putting a slightly positive spin on it: “When we’ve had outages in the past, it’s always led to us building new, more resilient systems.”
Follow TechRadar on Google News and add us as a preferred source to get our expert news, reviews and opinions in your feeds. Be sure to click the Follow button!
And of course you can too follow TechRadar on TikTok for news, reviews, video unboxings, and get regular updates from us on WhatsApp also.
