An obscure service provider briefly broke the internet Tuesday. It could happen again
A giant chunk of the internet briefly broke on Tuesday because of an outage at a company most people have probably never heard of before.
Reddit, CNN, Target, Amazon, a UK government website and countless others all went dark after a technical issue at cloud service provider Fastly.
Although the outage was short-lived, it served as a jarring reminder of the internet’s fragility. More than that, at a time when concerns are growing about cyber risks to critical physical US infrastructure, the Fastly outage may raise alarms about risks to our digital infrastructure, too.
Nearly all websites rely on a service provider like Fastly — which runs what’s called a “content delivery network” or CDN (we’ll get into what that means later on) — as a layer between internet users and the servers where their content is hosted. The problem: There are only a small handful of CDN operators. If one of them goes down — whether because of a benign software bug, as in Fastly’s case, or a cyberattack — huge swaths of the internet could go with it.
“Absolutely the biggest centralized point on the internet is these CDNs,” making them a potential target for cybercriminals or government actors, said Nick Merrill, research fellow at UC Berkeley’s Center for Long-Term Cybersecurity.
Utilities, social media platforms, news organizations, financial services, government agencies and more rely on CDNs like Fastly to operate their websites. Although Fastly was able to restore its service quickly, one can imagine problematic future scenarios if the resolution is slower.
“The problem with the internet is it’s always there until it isn’t,” said former Microsoft Chief Technology Officer David Vaskevitch, who now runs photo storage service Mylio. “For a system with so many interconnected parts, it’s not always reliable. Any one fragile part can bring it down.”
Even before this week’s outage, internet infrastructure experts have been ringing the alarm about concentration in the CDN space, where the small number of major providers could make for big targets for an attack.
What is a CDN?
For websites to load and run as quickly as we expect them to, they need to have computing power located physically close — at least relatively — to the people wanting to access them.
That’s why companies like Fastly exist. Fastly’s “content delivery network” is essentially a collection of “cloud” servers distributed across various geographic locations where websites can store content in close proximity to their users. This makes it possible for apps and websites to load within seconds and enables high quality streaming. It also saves huge amounts of energy.
CDNs play a crucial security role by preventing so-called “distributed denial-of-service” attacks, where bad actors send tons of requests to access a website in an effort to overwhelm its systems and shut it down.
“They’re indispensable infrastructure,” Merrill said.
The catch is that so many websites — big and small — use CDNs as a layer between users and the servers where their content lives that when a CDN goes down, much of the internet can go with it. In Tuesday’s case, a software bug that appeared as part of a normal update briefly took out around 85% of Fastly’s network, the company said.
And it’s not just CDNs. Amazon Web Services, a cloud computing service that supports numerous popular websites, has also experienced outages that end up taking down large chunks of the internet.
The risk
With any technology, occasional failures and outages are inevitable.
“There is no error-free internet, so the measure of success is how quickly a major internet firm like Fastly can recover from a rare outage like this,” said Doug Madory, director of internet analysis at network analytics firm Kentik.
Fastly detected Tuesday’s issue “within one minute,” and within less than an hour, 95% of its network was operating normally, senior vice president of engineering and infrastructure Nick Rockwell said in a blog post.
The bigger problem with the internet’s huge reliance on just a few CDN’s is the possibility that they become the target of an attack, Merrill said. He also worries about a potential government order dictating what such companies can and can’t provide support for, which could amount to government censorship of the internet.
Fastly is actually one of the smaller players in the CDN market. The largest is Cloudflare, which supports around 25 million internet properties including county websites, national ministries of health and corporate giants like IBM and Shopify. In 2019, Cloudflare was briefly in the spotlight after blocking support for 8Chan, making it difficult for the controversial online message board site to stay online.
To be sure, CDNs have backup protections in place and websites can contract with more than one CDN operator in case of failures. Most of the time, an outage will be like Tuesday’s — a temporary inconvenience. And websites could still appear online without a CDN, they’d just load slowly and be more at risk of cyberattacks.
But experts say there is still a risk that a bigger player like Cloudflare is targeted, or that multiple CDNs are hit at once.
“Worst case, it’s going to be an attack on Cloudflare,” Merrill said. “The Russian government or the Chinese government is going to take down Cloudflare and it’s going to break the internet.”
The solution, he said, could be antitrust regulation of the industry — similar to the regulatory pressure facing more consumer-facing tech companies — or promoting the growth of more CDN alternatives.
“People are really concerned rightly about antitrust issues in the tech space” Merrill said. “I don’t think that CDNs are as visible to people, but they’re probably the most important part of the core internet infrastructure that’s been privatized and centralized.”
