Comments

• Reply 1 of 18

  buttesilver Posts: 41member

  October 4, 2021 4:40PM

  Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

  
• Reply 2 of 18

  pulseimages Posts: 600member

  October 4, 2021 4:53PM

  It’s a giant PITA!

  
• Reply 3 of 18

  robin huber Posts: 3,958member

  October 4, 2021 5:12PM

  Thought it was my lousy Spectrum service. 
• Reply 4 of 18

  Xed Posts: 2,543member

  October 4, 2021 5:38PM

  buttesilver said:

  Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

  Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
  

  
• Reply 5 of 18

  zimmie Posts: 651member

  October 4, 2021 6:04PM

  It's actually a BGP issue. They advertise around 5000 prefixes into public BGP. A few hours ago, they stopped advertising around a thousand of those prefixes. It seems the prefixes they are no longer advertising contain most (possibly all) of their authoritative DNS servers. Even though a lot of their systems are still up, nothing can get to the DNS servers to find them.
  
  For extra fun, their internal communication platform is a separate instance of Facebook ... which is also inaccessible. People are speculating it's an attack, but if you're making a change remotely, and it takes out both your remote access and your ability to coordinate with others on the team, four hours or more is commonly the best case recovery time.
  
  Edit: Just heard (and second-sourced) that their badge access to buildings also isn't working. They take physical security pretty seriously. You can't just call a locksmith to pick the lock. It's looking likely this will last 12+ hours. I really feel for their IT operators.
  

  edited October 2021
• Reply 6 of 18

  ITGUYINSD Posts: 515member

  October 4, 2021 6:05PM

  robin huber said:

  Thought it was my lousy Spectrum service. 

  Funny enough, Specrum had a major outage this morning that covered at least most of San Diego, maybe even further north.  All my clients with Spectrum were OFFLINE.

  
• Reply 7 of 18

  docno42 Posts: 3,755member

  October 4, 2021 6:12PM

  pulseimages said:

  It’s a giant PITA!

  I'd say the overall quality of the Internet improves with each of these outages.  To each their own 

  
• Reply 8 of 18

  baconstang Posts: 1,105member

  October 4, 2021 6:49PM

  I would have never known had I not read it here.
  
  Maybe that is why the air is clearer today...

  
• Reply 9 of 18

  sflocal Posts: 6,093member

  October 4, 2021 7:06PM

  Keeping the Internet running is not an easy task.  I find it interesting how people will complain and whine when the Internet has a hiccup for a day, considering how well it runs for the remaining year.  

  
• Reply 10 of 18

  lkrupp Posts: 10,557member

  October 4, 2021 7:36PM

  sflocal said:

  Keeping the Internet running is not an easy task.  I find it interesting how people will complain and whine when the Internet has a hiccup for a day, considering how well it runs for the remaining year.  

  Louis CK Everything Is Amazing And Nobody Is Happy

  
  
  

  
• Reply 11 of 18

  verne arase Posts: 460member

  October 4, 2021 8:27PM

  Ugh ... DNS problems.
  
  I've experienced DNS problems - heck I caused DNS problems.
  
  Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.
  
  DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.
  
  Once burned, you avoid making changes as much as possible 😥.

  
• Reply 12 of 18

  Xed Posts: 2,543member

  October 4, 2021 9:07PM

  verne arase said:

  Ugh ... DNS problems.
  
  I've experienced DNS problems - heck I caused DNS problems.
  
  Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.
  
  DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.
  
  Once burned, you avoid making changes as much as possible 😥.

  Wait until you have to deal with a BGP table error. Oopsie.
  
  I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.

  
• Reply 13 of 18

  zimmie Posts: 651member

  October 4, 2021 9:24PM

  Xed said:

  verne arase said:

  Ugh ... DNS problems.
  
  I've experienced DNS problems - heck I caused DNS problems.
  
  Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.
  
  DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.
  
  Once burned, you avoid making changes as much as possible 😥.

  Wait until you have to deal with a BGP table error. Oopsie.
  
  I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.

  Almost definitely not an attack. This has all the hallmarks of a company overconfident in its intelligence designing itself into a pit. "We have a reliable system, so let's use it for internal discussion. Oh, and let's use it for badge access. And automation is the future, so let's execute all the changes without human intervention."
  
  When I was more involved in business IT, I had to constantly fight against people creating dependency cycles. That's the kind of situation where A has to be working for B to work, B has to be working for C to work, and C has to be working for A to work. For example, your VM environment has to be working for your password vault server to work, and your password vault server stores your passwords for fixing the VM environment if it dies. "Oh, but we never have problems with the VM environment!" Until somebody removes a LUN from your SAN and kernel panics all your VM hosts, and now the outage is 6 hours instead of 30 minutes.

  
• Reply 14 of 18

  buttesilver Posts: 41member

  October 4, 2021 10:29PM

  Xed said:

  buttesilver said:

  Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

  Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
  

  Did I hurt your butt, Xed?

  
• Reply 15 of 18

  Xed Posts: 2,543member

  October 4, 2021 11:29PM

  zimmie said:

  Xed said:

  verne arase said:

  Ugh ... DNS problems.
  
  I've experienced DNS problems - heck I caused DNS problems.
  
  Taught me a lot about default TTLs, inserting manual TTLs, and changing them well before initiating changes.
  
  DNS for infrastructure is like Social Security for politicians or the third rail for transit trains.
  
  Once burned, you avoid making changes as much as possible 😥.

  Wait until you have to deal with a BGP table error. Oopsie.
  
  I can't wait to read about what happened. It's looking like a deliberate hack, not an accident.

  Almost definitely not an attack. This has all the hallmarks of a company overconfident in its intelligence designing itself into a pit. "We have a reliable system, so let's use it for internal discussion. Oh, and let's use it for badge access. And automation is the future, so let's execute all the changes without human intervention."
  
  When I was more involved in business IT, I had to constantly fight against people creating dependency cycles. That's the kind of situation where A has to be working for B to work, B has to be working for C to work, and C has to be working for A to work. For example, your VM environment has to be working for your password vault server to work, and your password vault server stores your passwords for fixing the VM environment if it dies. "Oh, but we never have problems with the VM environment!" Until somebody removes a LUN from your SAN and kernel panics all your VM hosts, and now the outage is 6 hours instead of 30 minutes.

  I hope you're right, but from experience making an error in a BPG table is easily done even if fairly uncommon due to the amount of damage it can cause, but wiping entire tables is very, very rare. You have to either mess up really, really bad or it has to be deliberate.

  
• Reply 16 of 18

  Xed Posts: 2,543member

  October 4, 2021 11:33PM

  buttesilver said:

  Xed said:

  buttesilver said:

  Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

  Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
  

  Did I hurt your butt, Xed?

  Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)

  
• Reply 17 of 18

  buttesilver Posts: 41member

  October 5, 2021 12:16AM

  Xed said:

  buttesilver said:

  Xed said:

  buttesilver said:

  Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

  Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
  

  Did I hurt your butt, Xed?

  Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)

  Coincidentally, a number of whistleblowers have come out this week against FB.  But no, of course it's all UFO's to think they're scrubbing data.  Instead of trying to impress someone (don't know who) with your knowledge (or not) of BGP tables, open your mind.  If you don't think this sh*t happens, then you're spending too much time with tables and not enough with actual humans.

  
• Reply 18 of 18

  Xed Posts: 2,543member

  October 5, 2021 3:16AM

  buttesilver said:

  Xed said:

  buttesilver said:

  Xed said:

  buttesilver said:

  Can you say data scrub?  Yeah, that's all it is.  Time to revamp the culling of info to meet the narratives.

  Even when your conspiracy gateways are offline you're still finding a way to keep the momentum going.
  

  Did I hurt your butt, Xed?

  Interesting comment coming from a guy claiming that Facebook stopped access to three of its platforms so it could delete data. How exactly does bring excessive attention to itself help hide attention to itself? And what exactly was in those BGP tables that Zuckerberg didn't want others to know? (Hint: I know the answer and it's not very exciting.)

  Coincidentally, a number of whistleblowers have come out this week against FB.  But no, of course it's all UFO's to think they're scrubbing data.  Instead of trying to impress someone (don't know who) with your knowledge (or not) of BGP tables, open your mind.  If you don't think this sh*t happens, then you're spending too much time with tables and not enough with actual humans.

  You don't scrub data servers by hacking the BGP routers of ISPs. That would be an even bigger crime.
  
  Did you forget that Facebook controls their own servers? If they wanted to erase contents (as well as backups) they could do so without bringing down their revenue stream in the process. Talk about being a stupid criminal. I don't like FB, but I do give them more credit at skullduggery than you apparently do.