Facebook says 'faulty configuration change' to blame for 6-hour outage

2»

Comments

  • Reply 21 of 25
    zimmiezimmie Posts: 651member
    dewme said:
    zimmie said:
    dewme said:
    The concept behind Facebook is glorious if all people were nice. But all people aren’t nice. Some people are evil. Facebook has found a way to monetize evil and reap incalculable financial rewards from doing so. But as many others have said, if Facebook wasn’t doing it someone else would. The social media genie can never be put back in the bottle.
    Is it, though? The concept behind Facebook was to rate female Harvard students' hotness.

    dewme said:
    For six hours we were able to have meaningful in person conversations.   Maybe we need to start a GoFundMe for the poor IT guy that blew it today?  It was probably his last day.  
    I think that IT guy will never make that mistake again, so he would be a great employee now.

    The one forbidden word for surgeons and network engineers is:   "Ooops!"

    I can only imagine the "conversations" going on today between FaceBook engineers, managers, etc...
    The blame game will have reached new, unheard of levels by now....

    It does seen odd that they would not employ some sort of checkpointing scheme on their configuration database to allow them to roll back to the last known good state. This is a very common technique for high availability systems and even some individual products, e.g., take a snapshot of the configuration settings before performing a software or firmware update. While I have zero love for Facebook, I'm sure that its stakeholders don't appreciate the financial losses incurred during the protracted downtime.
    They do have the ability to undo changes ... at the level of the management platform. When the management platform can no longer reach the device to manage it, it needs manual intervention to restore that communication. Then the configuration can be restored to what it was before the change. They also lost external access to the management platform, but that's simple enough to fix (if somebody doesn't confirm the change, roll it back automatically). The access from the management to the managed devices is more difficult.

    Good information. That makes sense. So they basically severed the primary communication link between the management platform and the managed devices. So what they also needed was a remote out-of-band communication channel, or what they probably ended up doing, a bunch of engineers with laptops and a bunch of physical security keys to bypass the badge readers protecting the servers. Interesting cascading failure mode.
    That's just it. They had an out-of-band channel to connect to the devices and fix them ... but that channel became unreachable when all the DNS servers took themselves offline. They depended too much on systems that "can't ever all go down at once", only to be caught flat-footed when they all went down at once.
    muthuk_vanalingamwatto_cobra
  • Reply 22 of 25
    JapheyJaphey Posts: 1,767member
    sflocal said:
    Fred257 said:
    This outage was done on purpose. On Sunday evening 60 minutes did a major story on Facebook from an actual whistleblower.  To get that story out of the media what better way than to create a major diversion.  Zuckerberg is a covert psychopath narcissist and I know how exactly he thinks because I have a lifetime experience with them.  Zuckerberg is a sick person who extracts energy and money from destroying our personal lives with his extraction tool called Facebook. 
    You should consider a better-quality tinfoil for your next hat.  The silliness factor of your conspiracy-post is high.
    The problem with automatically labeling everything a conspiracy theory and flippantly shaming those who propose any theory in opposition of your own is that when an actual conspiracy does happen, you won’t even know it until it is too late. This isn’t exactly nazi bases on Antarctica-level stuff. His theory is both logical and believable. Do you really think that Zuck isn’t capable of such a misdirect? By next week, most people will only vaguely (if at all) remember the whistleblower story, but nearly all of them will vividly remember the whole day that Facebook was down. 
    watto_cobra
  • Reply 23 of 25
    lkrupplkrupp Posts: 10,557member
    sflocal said:
    Fred257 said:
    This outage was done on purpose. On Sunday evening 60 minutes did a major story on Facebook from an actual whistleblower.  To get that story out of the media what better way than to create a major diversion.  Zuckerberg is a covert psychopath narcissist and I know how exactly he thinks because I have a lifetime experience with them.  Zuckerberg is a sick person who extracts energy and money from destroying our personal lives with his extraction tool called Facebook. 
    You should consider a better-quality tinfoil for your next hat.  The silliness factor of your conspiracy-post is high.
    And probably an anti-vaxxer to boot.
  • Reply 24 of 25
    GeorgeBMacGeorgeBMac Posts: 11,421member
    docno42 said:
    GeorgeBMac said:
    Or being used to spread misinformation and as a forum for terrorists to recruit more terrorists and plan their attacks.
    lol - the new "but think of the children". 

    Yes Zuck, save us from ourselves!  

    Pathetic...

    Yes, that was the misinformation from Russia that helped turn an election and those were internal terrorists I was speaking of.  The ones who planned & organized their attack on our Capitol openly on social media.

    And I agree:  anybody who accepts -- or supports it --  is pathetic.
Sign In or Register to comment.