Facebook, Instagram, and other services seeing widespread outage [u]

AppleInsider

Facebook and its various services -- including Instagram and WhatsApp -- are currently experiencing a widespread outage that is seemingly being caused by DNS problems.

Credit: Brett Jordan/Unsplash

Reports first started surfacing around 11:30 a.m. Eastern Time (8:30 a.m. Pacific), according to Down Detector. The outage appears widespread, with 126,352 reports of an issue by 11:58 a.m. It isn't just Facebook's main site that is being affected, either. Instagram, WhatsApp, Messenger, and Oculus VR appear down, too.

The social media giant acknowledged the issue around 12:16 p.m., stating that they're "working to get things back to normal as quickly as possible."

We're aware that some people are having trouble accessing Facebook app. We're working to get things back to normal as quickly as possible, and we apologize for any inconvenience.

-- Facebook App (@facebookapp)

October 4, 2021

The exact cause of the issue isn't known, but seems to be related to DNS. Around noon Eastern time on Monday, a Cloudflare executive said that it appears that Facebook's BGP routes have been "withdrawn from the internet."

Update: Facebook's services returned after an approximately six-hour outage.

Read on AppleInsider

buttesilver

Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

pulseimages

It’s a giant PITA!

robin huber

Thought it was my lousy Spectrum service. 

Xed

buttesilver said:

Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.

zimmie

It's actually a BGP issue. They advertise around 5000 prefixes into public BGP. A few hours ago, they stopped advertising around a thousand of those prefixes. It seems the prefixes they are no longer advertising contain most (possibly all) of their authoritative DNS servers. Even though a lot of their systems are still up, nothing can get to the DNS servers to find them.

For extra fun, their internal communication platform is a separate instance of Facebook ... which is also inaccessible. People are speculating it's an attack, but if you're making a change remotely, and it takes out both your remote access and your ability to coordinate with others on the team, four hours or more is commonly the best case recovery time.

Edit: Just heard (and second-sourced) that their badge access to buildings also isn't working. They take physical security pretty seriously. You can't just call a locksmith to pick the lock. It's looking likely this will last 12+ hours. I really feel for their IT operators.

ITGUYINSD

robin huber said:

Thought it was my lousy Spectrum service. 

Funny enough, Specrum had a major outage this morning that covered at least most of San Diego, maybe even further north.  All my clients with Spectrum were OFFLINE.

docno42

pulseimages said:

It’s a giant PITA!

I'd say the overall quality of the Internet improves with each of these outages.  To each their own 

baconstang

I would have never known had I not read it here.

Maybe that is why the air is clearer today...

sflocal

Keeping the Internet running is not an easy task.  I find it interesting how people will complain and whine when the Internet has a hiccup for a day, considering how well it runs for the remaining year.  

lkrupp

sflocal said:

Keeping the Internet running is not an easy task.  I find it interesting how people will complain and whine when the Internet has a hiccup for a day, considering how well it runs for the remaining year.  

Louis CK Everything Is Amazing And Nobody Is Happy

verne arase

Ugh ... DNS problems.

I've experienced DNS problems - heck I caused DNS problems.

Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.

DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.

Once burned, you avoid making changes as much as possible 😥.

Xed

verne arase said:

Ugh ... DNS problems.

I've experienced DNS problems - heck I caused DNS problems.

Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.

DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.

Once burned, you avoid making changes as much as possible 😥.

Wait until you have to deal with a BGP table error. Oopsie.

I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.

zimmie

Xed said:

verne arase said:

Ugh ... DNS problems.

I've experienced DNS problems - heck I caused DNS problems.

Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.

DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.

Once burned, you avoid making changes as much as possible 😥.

Wait until you have to deal with a BGP table error. Oopsie.

I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.

Almost definitely not an attack. This has all the hallmarks of a company overconfident in its intelligence designing itself into a pit. "We have a reliable system, so let's use it for internal discussion. Oh, and let's use it for badge access. And automation is the future, so let's execute all the changes without human intervention."

When I was more involved in business IT, I had to constantly fight against people creating dependency cycles. That's the kind of situation where A has to be working for B to work, B has to be working for C to work, and C has to be working for A to work. For example, your VM environment has to be working for your password vault server to work, and your password vault server stores your passwords for fixing the VM environment if it dies. "Oh, but we never have problems with the VM environment!" Until somebody removes a LUN from your SAN and kernel panics all your VM hosts, and now the outage is 6 hours instead of 30 minutes.

buttesilver

Xed said:

buttesilver said:

Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.

Did I hurt your butt, Xed?

Xed

zimmie said:

Xed said:

verne arase said:

Ugh ... DNS problems.

I've experienced DNS problems - heck I caused DNS problems.

Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.

DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.

Once burned, you avoid making changes as much as possible 😥.

Wait until you have to deal with a BGP table error. Oopsie.

I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.

Almost definitely not an attack. This has all the hallmarks of a company overconfident in its intelligence designing itself into a pit. "We have a reliable system, so let's use it for internal discussion. Oh, and let's use it for badge access. And automation is the future, so let's execute all the changes without human intervention."

When I was more involved in business IT, I had to constantly fight against people creating dependency cycles. That's the kind of situation where A has to be working for B to work, B has to be working for C to work, and C has to be working for A to work. For example, your VM environment has to be working for your password vault server to work, and your password vault server stores your passwords for fixing the VM environment if it dies. "Oh, but we never have problems with the VM environment!" Until somebody removes a LUN from your SAN and kernel panics all your VM hosts, and now the outage is 6 hours instead of 30 minutes.

I hope you're right, but from experience making an error in a BPG table is easily done even if fairly uncommon due to the amount of damage it can cause, but wiping entire tables is very, very rare. You have to either mess up really, really bad or it has to be deliberate.

Xed

buttesilver said:

Xed said:

buttesilver said:

Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.

Did I hurt your butt, Xed?

Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)

buttesilver

Xed said:

buttesilver said:

Xed said:

buttesilver said:

Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.

Did I hurt your butt, Xed?

Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)

Coincidentally, a number of whistleblowers have come out this week against FB.  But no, of course it's all UFO's to think they're scrubbing data.  Instead of trying to impress someone (don't know who) with your knowledge (or not) of BGP tables, open your mind.  If you don't think this sh*t happens, then you're spending too much time with tables and not enough with actual humans.

Xed

buttesilver said:

Xed said:

buttesilver said:

Xed said:

buttesilver said:

Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.

Did I hurt your butt, Xed?

Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)

Coincidentally, a number of whistleblowers have come out this week against FB.  But no, of course it's all UFO's to think they're scrubbing data.  Instead of trying to impress someone (don't know who) with your knowledge (or not) of BGP tables, open your mind.  If you don't think this sh*t happens, then you're spending too much time with tables and not enough with actual humans.

You don't scrub data servers by hacking the BGP routers of ISPs. That would be an even bigger crime.

Did you forget that Facebook controls their own servers? If they wanted to erase contents (as well as backups) they could do so without bringing down their revenue stream in the process. Talk about being a stupid criminal. I don't like FB, but I do give them more credit at skullduggery than you apparently do.