Facebook went down yesterday, which caused quite a bit of excitement over the internet. I was originally hoping to draft some observations from a technical standpoint, but I read this article from Cloudflare and felt it was better than what I was working on.
Instead, I’m going to keep it less technical and speak to my thoughts on where the internet should go from here.
It occurred to me that many people in my slack group were surprised that instragram, facebook, and whatsapp were all down – despite being owned by the same entity. I understand the idea that all these things should be redundant, have millions of backups of all varieties, and be secured to a point where this sort of thing could never happen. While this was the case to some degree, I have three main takeaways from how things shook down.
Every company, no matter how large, is incentivized to cut costs when they can.
“Every lock made by man can be broken by man”
DNS is the worst
This particular issue, like so many IT issues, really came down to DNS.
Facebook’s statement clarified that it was a configuration change on backbone routers that caused the issue. (see clipping of statement below)
“Our engineering teams have learned that configuration changes on the backbone routers that coordinate network traffic between our data centers caused issues that interrupted this communication. This disruption to network traffic had a cascading effect on the way our data centers communicate, bringing our services to a halt.”
This has really left me fascinated with how exactly a configuration change like this can happen in the first place. I’m not going to dive into heavy speculation, because as I am sure you can tell by now I like to keep this blog pretty vanilla, but this seems like the most radical accident ever. My hope is that we will get more information in the coming days that clarify what exactly happened.
I am no fan of Facebook – although I do find it’s group functionality very useful. Many people on my Twitter were cheering at the collapse of Facebook, Instagram, and Whatsapp. I mostly agree that it would be a net positive to have these platforms all eradicated, however, I think this is a good learning experience for everyone that does use these platforms as a photo album, a password authentication, or the sole means to communicate with loved ones. My genuine first thought when I heard Facebook was completely down was “What are all of those people that use Facebook to log into all of their other apps going to do?”. That thought was heavily overshadowed by the cheers for Zuckerberg’s collapse.
Never rely on just one method for getting necessary things accomplished – make your life redundant, and Fortify Your Data.
Sources:
https://blog.cloudflare.com/october-2021-facebook-outage/
https://engineering.fb.com/2021/10/04/networking-traffic/outage/