Facebook, Instagram, and other services seeing widespread outage [u]

Posted:
in General Discussion edited October 4
Facebook and its various services -- including Instagram and WhatsApp -- are currently experiencing a widespread outage that is seemingly being caused by DNS problems.

Credit: Brett Jordan/Unsplash
Credit: Brett Jordan/Unsplash


Reports first started surfacing around 11:30 a.m. Eastern Time (8:30 a.m. Pacific), according to Down Detector. The outage appears widespread, with 126,352 reports of an issue by 11:58 a.m. It isn't just Facebook's main site that is being affected, either. Instagram, WhatsApp, Messenger, and Oculus VR appear down, too.

The social media giant acknowledged the issue around 12:16 p.m., stating that they're "working to get things back to normal as quickly as possible."

We're aware that some people are having trouble accessing Facebook app. We're working to get things back to normal as quickly as possible, and we apologize for any inconvenience.

-- Facebook App (@facebookapp)


The exact cause of the issue isn't known, but seems to be related to DNS. Around noon Eastern time on Monday, a Cloudflare executive said that it appears that Facebook's BGP routes have been "withdrawn from the internet."

Update: Facebook's services returned after an approximately six-hour outage.

Read on AppleInsider

Comments

  • Reply 1 of 18
    Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.
    robin huberdocno42jahbladeboltsfan17killroywatto_cobra
  • Reply 2 of 18
    It’s a giant PITA!
    fotoformat
  • Reply 3 of 18
    robin huberrobin huber Posts: 3,517member
    Thought it was my lousy Spectrum service. 
  • Reply 4 of 18
    XedXed Posts: 1,106member
    Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.
    Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
    djames4242TRAGbaconstangkillroyStrangeDays
  • Reply 5 of 18
    zimmiezimmie Posts: 568member
    It's actually a BGP issue. They advertise around 5000 prefixes into public BGP. A few hours ago, they stopped advertising around a thousand of those prefixes. It seems the prefixes they are no longer advertising contain most (possibly all) of their authoritative DNS servers. Even though a lot of their systems are still up, nothing can get to the DNS servers to find them.

    For extra fun, their internal communication platform is a separate instance of Facebook ... which is also inaccessible. People are speculating it's an attack, but if you're making a change remotely, and it takes out both your remote access and your ability to coordinate with others on the team, four hours or more is commonly the best case recovery time.

    Edit: Just heard (and second-sourced) that their badge access to buildings also isn't working. They take physical security pretty seriously. You can't just call a locksmith to pick the lock. It's looking likely this will last 12+ hours. I really feel for their IT operators.
    edited October 4 elijahgdewmeappleinsideruserfastasleepkillroywatto_cobra
  • Reply 6 of 18
    Thought it was my lousy Spectrum service. 
    Funny enough, Specrum had a major outage this morning that covered at least most of San Diego, maybe even further north.  All my clients with Spectrum were OFFLINE.
    watto_cobra
  • Reply 7 of 18
    docno42docno42 Posts: 3,660member
    It’s a giant PITA!
    I'd say the overall quality of the Internet improves with each of these outages.  To each their own  :p
    TRAGbaconstangelijahgbuttesilverkillroywatto_cobra
  • Reply 8 of 18
    I would have never known had I not read it here.

    Maybe that is why the air is clearer today...
    docno42lkruppwilliamlondondewmewatto_cobra
  • Reply 9 of 18
    sflocalsflocal Posts: 5,820member
    Keeping the Internet running is not an easy task.  I find it interesting how people will complain and whine when the Internet has a hiccup for a day, considering how well it runs for the remaining year.  
    williamlondonDAalsethwatto_cobra
  • Reply 10 of 18
    lkrupplkrupp Posts: 9,628member
    sflocal said:
    Keeping the Internet running is not an easy task.  I find it interesting how people will complain and whine when the Internet has a hiccup for a day, considering how well it runs for the remaining year.  

    Louis CK Everything Is Amazing And Nobody Is Happy



    jas99Xedwatto_cobra
  • Reply 11 of 18
    Ugh ... DNS problems.

    I've experienced DNS problems - heck I caused DNS problems.

    Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.

    DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.

    Once burned, you avoid making changes as much as possible 😥.
    watto_cobra
  • Reply 12 of 18
    XedXed Posts: 1,106member
    Ugh ... DNS problems.

    I've experienced DNS problems - heck I caused DNS problems.

    Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.

    DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.

    Once burned, you avoid making changes as much as possible 😥.
    Wait until you have to deal with a BGP table error. Oopsie.

    I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.
    watto_cobra
  • Reply 13 of 18
    zimmiezimmie Posts: 568member
    Xed said:
    Ugh ... DNS problems.

    I've experienced DNS problems - heck I caused DNS problems.

    Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.

    DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.

    Once burned, you avoid making changes as much as possible 😥.
    Wait until you have to deal with a BGP table error. Oopsie.

    I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.
    Almost definitely not an attack. This has all the hallmarks of a company overconfident in its intelligence designing itself into a pit. "We have a reliable system, so let's use it for internal discussion. Oh, and let's use it for badge access. And automation is the future, so let's execute all the changes without human intervention."

    When I was more involved in business IT, I had to constantly fight against people creating dependency cycles. That's the kind of situation where A has to be working for B to work, B has to be working for C to work, and C has to be working for A to work. For example, your VM environment has to be working for your password vault server to work, and your password vault server stores your passwords for fixing the VM environment if it dies. "Oh, but we never have problems with the VM environment!" Until somebody removes a LUN from your SAN and kernel panics all your VM hosts, and now the outage is 6 hours instead of 30 minutes.
    killroywatto_cobra
  • Reply 14 of 18
    Xed said:
    Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.
    Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
    Did I hurt your butt, Xed?
    killroywatto_cobra
  • Reply 15 of 18
    XedXed Posts: 1,106member
    zimmie said:
    Xed said:
    Ugh ... DNS problems.

    I've experienced DNS problems - heck I caused DNS problems.

    Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.

    DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.

    Once burned, you avoid making changes as much as possible 😥.
    Wait until you have to deal with a BGP table error. Oopsie.

    I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.
    Almost definitely not an attack. This has all the hallmarks of a company overconfident in its intelligence designing itself into a pit. "We have a reliable system, so let's use it for internal discussion. Oh, and let's use it for badge access. And automation is the future, so let's execute all the changes without human intervention."

    When I was more involved in business IT, I had to constantly fight against people creating dependency cycles. That's the kind of situation where A has to be working for B to work, B has to be working for C to work, and C has to be working for A to work. For example, your VM environment has to be working for your password vault server to work, and your password vault server stores your passwords for fixing the VM environment if it dies. "Oh, but we never have problems with the VM environment!" Until somebody removes a LUN from your SAN and kernel panics all your VM hosts, and now the outage is 6 hours instead of 30 minutes.
    I hope you're right, but from experience making an error in a BPG table is easily done even if fairly uncommon due to the amount of damage it can cause, but wiping entire tables is very, very rare. You have to either mess up really, really bad or it has to be deliberate.
    watto_cobra
  • Reply 16 of 18
    XedXed Posts: 1,106member
    Xed said:
    Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.
    Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
    Did I hurt your butt, Xed?
    Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)
    watto_cobra
  • Reply 17 of 18
    Xed said:
    Xed said:
    Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.
    Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
    Did I hurt your butt, Xed?
    Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)
    Coincidentally, a number of whistleblowers have come out this week against FB.  But no, of course it's all UFO's to think they're scrubbing data.  Instead of trying to impress someone (don't know who) with your knowledge (or not) of BGP tables, open your mind.  If you don't think this sh*t happens, then you're spending too much time with tables and not enough with actual humans.
    watto_cobra
  • Reply 18 of 18
    XedXed Posts: 1,106member
    Xed said:
    Xed said:
    Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.
    Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
    Did I hurt your butt, Xed?
    Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)
    Coincidentally, a number of whistleblowers have come out this week against FB.  But no, of course it's all UFO's to think they're scrubbing data.  Instead of trying to impress someone (don't know who) with your knowledge (or not) of BGP tables, open your mind.  If you don't think this sh*t happens, then you're spending too much time with tables and not enough with actual humans.
    You don't scrub data servers by hacking the BGP routers of ISPs. That would be an even bigger crime.

    Did you forget that Facebook controls their own servers? If they wanted to erase contents (as well as backups) they could do so without bringing down their revenue stream in the process. Talk about being a stupid criminal. I don't like FB, but I do give them more credit at skullduggery than you apparently do.
    muthuk_vanalingamwatto_cobra
Sign In or Register to comment.