Global chaos erupts as Windows security update goes bad

AppleInsider

The vast majority of corporate IT worldwide is struggling on Friday morning, with things as mundane as point-of-purchase, and as complex as flight management not working because of a bad Windows security patch by security firm CrowdStrike.

An unknown number of Apple Pay terminals are affected by the Microsoft outage worldwide

While the failure is confined to Windows systems, it's significantly worse than previous Microsoft outages, because of the scale. American Airlines, Delta, and United, each grounded all aircraft, according to BBC News, TV stations including MTV, VH1, CMT, Sky News, and ABC News Australia went off air.

What's directly affecting Apple users is that there are now reports of supermarkets around the world having problems accepting Apple Pay and other contactless payments. This will be because they are using Windows-based terminals, but it's not clear either how widespread this issue is, nor why it isn't affecting all users.

The outage was caused by a software update by security firm Crowdstrike. The company has issued a brief statement saying that it was one issue in an update, and that "this is not a security incident or cyberattack."

"CrowdStrike is actively working with customers impacted by a defect found in a single content update for Windows hosts," continues the statement. "Mac and Linux hosts are not impacted."

"The issue has been identified, isolated and a fix has been deployed," says the company.

Crowdstrike has not given a timescale for when the fix will be adequately rolled out worldwide. At present, the issue is continuing, and at time of writing, Apple Pay has been seeing a spike in outages, presumably because of it.



Read on AppleInsider

bluelightning

Surprised something like this would get past testing (assuming testing was done).  

A number of books have been written on the anticipated failure of complex systems, sometimes looking at telecommunications or power delivery.  

applezulu

So this is where we see letting third-party vendors have that level of access to the OS in order to provide security becomes a vulnerability in itself. 

avon b7

My scheduled video conference with HP has hit the skids. I was told at 8am this morning that IT support were working on the problem but I doubt they'll apply the machine-by-machine workaround offered by Crowdstrike which requires booting into Safe Mode and deleting a file.

It's touch and go for the meeting which is scheduled to start in 30 minutes. 

It is more likely they'll wait for the fix from Crowdstrike to flush through the systems. 

Global automated updates should be rolled out in phases to catch these glitches before they become wildfires. 

I see some airlines still have boarding cards that can be handwritten for the lucky ones who have been able to get off the ground.

EDIT: meeting postponed.

edited July 2024

avon b7

AppleZulu said:

So this is where we see letting third-party vendors have that level of access to the OS in order to provide security becomes a vulnerability in itself. 

These issues can affect first party vendors in exactly the same way. 

If your systems permeate the fabric of the internet (in the deployment sense), the scope for trouble is there but everyone (first or third party) should have resilience designed into their systems. 

Crowdstrike says this is the result of a botched update as opposed to a security or cyber attack situation but having it propagate so far and so quickly has to be looked at and resolved.

Any vendor could be hit by botched processes so that includes Apple, Meta, Google, Microsoft, Amazon Huawei etc in the deployment sense and the likes of Cloudfare in an infrastructure sense.

I suppose using a third party solution might even be favorable in some situations.

Luckily, these situations don't usually hit the headlines very often but some might ask 'how much of that is literally down to 'luck' itself?

edited July 2024

thesparkle

No sooner are Labour in power… /s

itguyinsd

The headline makes it sound like a Windows Update caused the issue, when in fact, the issue was a 3rd party software company.

Could have been worded better.  People are already blaming Windows...this headline doesn't help.

blastdoor

AppleZulu said:

So this is where we see letting third-party vendors have that level of access to the OS in order to provide security becomes a vulnerability in itself. 

And yet if Microsoft tried to take control themselves the EU would fine then eleventy bazillion dollars a day.

blastdoor

avon b7 said:

AppleZulu said:

So this is where we see letting third-party vendors have that level of access to the OS in order to provide security becomes a vulnerability in itself. 

These issues can affect first party vendors in exactly the same way. 

Sure, they “can,” but the important question is the probability that it happens. If first party reduces the probability, then first party control is the way to go.

across multiple industries, I think we are now seeing the advantages of vertical integration. Apple has been a champion of vertical integration for a long time, but now we are also seeing how formerly integrated companies are crashing and burning when they give up control of key product components — ie Boeing. 

cheesefreeze

AppleZulu said:

So this is where we see letting third-party vendors have that level of access to the OS in order to provide security becomes a vulnerability in itself. 

Nonsense. Good standards don’t let third parties to operate “on an OS level” since that access is not needed to begin with. In case of Apple and Microsoft they provide APIs and frameworks that other vendors can use. They stay within these boundaries (or there wouldn’t be a standard).

Also, you could argue the very opposite. Letting a few tech giants in control over protocols used world-wide means that when things go bad, everything goes bad. 

pema

So, what else is new? Or News? MS DOS, MS Windows, always the achilles heel of the tech world. But because Mr Gates forced the both down pc makers throats then it spread like a virus. For those who don't recall, you could build a PC, no quality control there. Get a motherboard, a breadbox, slap a display on top call it Joe's PC and you are in business. But, and this is the big but, hardware alone is no good without the software. So Mr Gates tells the pc makers, ok, folks you want to put MS DOS on there, fine there is a licensing fee per PC. And that trend continued with Windows. 
Apple was different the hardware and software came bundled together so they worked seamlessly. 
There was one time in Apple's history where they almost became extinct and that when Mr Pepsi Cola - some idiot that Steve Jobs hired from the soft drink company to manage the company - licensed the Apple OS and a builder in Texas I believe put together the hardware and then slapped on the Apple OS. That relationship lasted as long as a one-night stand. 
Long story short, Mr Pepsi Cola forced Steve Jobs out, Jobs created Next on top of Unix. Apple floundered, MS tipped $150 million into Apple, Steve came back resurrected the company on top of the Unix Shell with the Mac interface. And from the Bondi Blue Mac and the iPod and as they say the rest is history. Apple is, IMO, due to become a $5 Trillion company. 
Microsoft's O/S was some rubbish that Ms Gates purloined from a guy in Seattle. It has always been sheit. Full of security holes, bugs you name it. MS ought to have dumped the whole kit and caboodle, and become what they have proven to be good at, Cloud. My Nutella's specialty. Except good old Windows has opened up a breach into the cloud and scuttled the works. 
I am no big fan of Android but it is heads and shoulders above Windows. 
Perhaps MS needs to team up with Google and license Android to run their PCs and Azure to manage the Cloud. 
Otherwise there will be days like these. 
 

omasou

Well, now Tim can point to this incident next time ANY governments asks Apple to put loosen security or provide a backdoor.


Outside Apple: We need access to the secure enclave
Tim: No

Outside Apple: We need access to the the inner workings of Apple Pay
Tim: No

Outside Apple: We need access to ..., you're anti-competitive....
Tim: No we are NOT and this is why we lock down our systems and maintain such a high level of CONTROL. Now go pound sand.

edited July 2024

lettuce

avon b7 said:

My scheduled video conference with HP has hit the skids. I was told at 8am this morning that IT support were working on the problem but I doubt they'll apply the machine-by-machine workaround offered by Crowdstrike which requires booting into Safe Mode and deleting a file.

It's touch and go for the meeting which is scheduled to start in 30 minutes. 

It is more likely they'll wait for the fix from Crowdstrike to flush through the systems. 

Global automated updates should be rolled out in phases to catch these glitches before they become wildfires. 

I see some airlines still have boarding cards that can be handwritten for the lucky ones who have been able to get off the ground.

EDIT: meeting postponed.

The corrected update has already been rolled out. If the computer installed the wrong one it stopped working and the only way to fix it is to manually intervene. 

edited July 2024

avon b7

blastdoor said:

avon b7 said:

AppleZulu said:

So this is where we see letting third-party vendors have that level of access to the OS in order to provide security becomes a vulnerability in itself. 

These issues can affect first party vendors in exactly the same way. 

Sure, they “can,” but the important question is the probability that it happens. If first party reduces the probability, then first party control is the way to go.

across multiple industries, I think we are now seeing the advantages of vertical integration. Apple has been a champion of vertical integration for a long time, but now we are also seeing how formerly integrated companies are crashing and burning when they give up control of key product components — ie Boeing. 

I don't think the probability angle is applicable here because these things happen anywhere, at any time. First or third party vendors alike. They shouldn't have in the way this one did. 

Apple has famously deleted user data basically with a flick of a switch. That turned out to be pure slopiness. Again, it shouldn't happen but sometimes it does. 

The extent of an exploit, bug, glitch or whatever is always going to be dependent on the extent of the deployment platform. This one has had a tsunami effect because of that. 

It's probably completely untrue but some will gleefully proclaim that maybe it was just as well it was a third party because if it were first party (Microsoft) more such cases would be the order of the day. LOL. 

It seems Microsoft does a pretty good job with its corporate/server/cloud solutions but I have no idea if that is really the case of if it is all a house of cards waiting to collapse. 

hammeroftruth

Lettuce said:

avon b7 said:

My scheduled video conference with HP has hit the skids. I was told at 8am this morning that IT support were working on the problem but I doubt they'll apply the machine-by-machine workaround offered by Crowdstrike which requires booting into Safe Mode and deleting a file.

It's touch and go for the meeting which is scheduled to start in 30 minutes. 

It is more likely they'll wait for the fix from Crowdstrike to flush through the systems. 

Global automated updates should be rolled out in phases to catch these glitches before they become wildfires. 

I see some airlines still have boarding cards that can be handwritten for the lucky ones who have been able to get off the ground.

EDIT: meeting postponed.

The corrected update has already been rolled out. If the computer installed the wrong one it stopped working and the only way to fix it is to manually intervene. 

You mean follow Microsoft’s advice?

“Several reboots (as many as 15 have been reported) may be required, but overall feedback is that reboots are an effective troubleshooting step at this stage,"

avon b7

Pema said:

So, what else is new? Or News? MS DOS, MS Windows, always the achilles heel of the tech world. But because Mr Gates forced the both down pc makers throats then it spread like a virus. For those who don't recall, you could build a PC, no quality control there. Get a motherboard, a breadbox, slap a display on top call it Joe's PC and you are in business. But, and this is the big but, hardware alone is no good without the software. So Mr Gates tells the pc makers, ok, folks you want to put MS DOS on there, fine there is a licensing fee per PC. And that trend continued with Windows. 
Apple was different the hardware and software came bundled together so they worked seamlessly. 
There was one time in Apple's history where they almost became extinct and that when Mr Pepsi Cola - some idiot that Steve Jobs hired from the soft drink company to manage the company - licensed the Apple OS and a builder in Texas I believe put together the hardware and then slapped on the Apple OS. That relationship lasted as long as a one-night stand. 
Long story short, Mr Pepsi Cola forced Steve Jobs out, Jobs created Next on top of Unix. Apple floundered, MS tipped $150 million into Apple, Steve came back resurrected the company on top of the Unix Shell with the Mac interface. And from the Bondi Blue Mac and the iPod and as they say the rest is history. Apple is, IMO, due to become a $5 Trillion company. 
Microsoft's O/S was some rubbish that Ms Gates purloined from a guy in Seattle. It has always been sheit. Full of security holes, bugs you name it. MS ought to have dumped the whole kit and caboodle, and become what they have proven to be good at, Cloud. My Nutella's specialty. Except good old Windows has opened up a breach into the cloud and scuttled the works. 
I am no big fan of Android but it is heads and shoulders above Windows. 
Perhaps MS needs to team up with Google and license Android to run their PCs and Azure to manage the Cloud. 
Otherwise there will be days like these. 
 

To be fair, Classic Mac OS was never even an option for mission critical systems and lack of long term support from Apple was another issue, as was corporate support for its systems. 

In the mid nineties I was pushing to get Apple into education systems here (an area of 7 million people) but any deployment would hinge on local language support which Apple wasn't interested in providing. 

OSX brought POSIX compliance but by then it was impossible to compete with an entrenched Windows ecosystem and the illegal practices of Microsoft to lock customers into their systems. 

One potentially huge issue for Apple's rumoured car adventure was committment and long term software support. Apple has never been into either. 

dewme

CheeseFreeze said:

AppleZulu said:

So this is where we see letting third-party vendors have that level of access to the OS in order to provide security becomes a vulnerability in itself. 

Nonsense. Good standards don’t let third parties to operate “on an OS level” since that access is not needed to begin with. In case of Apple and Microsoft they provide APIs and frameworks that other vendors can use. They stay within these boundaries (or there wouldn’t be a standard).

Also, you could argue the very opposite. Letting a few tech giants in control over protocols used world-wide means that when things go bad, everything goes bad. 

Yes indeed. This is a prime example of what happens when any complex systems has a single point or common mode failure vulnerability. But that’s not the full extent of the problem either. This is also a process problem on at least two levels. The software vendor absolutely should have tested the update internally on customer representative system test beds before release. 

Secondly, the owners of the affected systems should have internal processes in place to verify that updates are first deployed on a small subset of managed systems, usually ones controlled by IT, before they are widely deployed on business critical systems. The latter process is why most companies who take security and uptime seriously do not allow end users to individually update corporate managed IT resources. All updates are verified and validated prior to making them available or pushing them out to individual machines and devices.

Lastly, I suppose we can add a third process issue too. Companies that place their critical business IT assets or infrastructure in the hands of a third party service provider without establishing internal guardrails, i.e., fully trusting that the service provider will save them from harm with no responsibility on their side, are operating at a higher level of risk. While some of these service providers do provide compensation for financial losses due to service disruptions, the company that owns the system is putting its own reputation on the line because their customers will only blame the company for disruptions, not the third party service provider. 

Subcontracting and outsourcing does not absolve a company from living up to their responsibilities. This is something Apple understands and has had to live with on many occasions, e.g., Foxconn and other contractors. As a matter of fact, one of the things that attracted me to Apple was the fact that they never abdicated responsibility for their products to anyone other than themselves. Anyone who’s lived in the Windows PC world knows all too well about the double sided finger pointing that goes on between system vendors and Microsoft when there’s a problem with your system. It used to be very common for the first step in trying to restore a messed up Windows machine was “Reinstall Windows.” Arggg!

edited July 2024

mike1

Pema said:

So, what else is new? Or News? MS DOS, MS Windows, always the achilles heel of the tech world. But because Mr Gates forced the both down pc makers throats then it spread like a virus. For those who don't recall, you could build a PC, no quality control there. Get a motherboard, a breadbox, slap a display on top call it Joe's PC and you are in business. But, and this is the big but, hardware alone is no good without the software. So Mr Gates tells the pc makers, ok, folks you want to put MS DOS on there, fine there is a licensing fee per PC. And that trend continued with Windows. 
Apple was different the hardware and software came bundled together so they worked seamlessly. 
There was one time in Apple's history where they almost became extinct and that when Mr Pepsi Cola - some idiot that Steve Jobs hired from the soft drink company to manage the company - licensed the Apple OS and a builder in Texas I believe put together the hardware and then slapped on the Apple OS. That relationship lasted as long as a one-night stand. 
Long story short, Mr Pepsi Cola forced Steve Jobs out, Jobs created Next on top of Unix. Apple floundered, MS tipped $150 million into Apple, Steve came back resurrected the company on top of the Unix Shell with the Mac interface. And from the Bondi Blue Mac and the iPod and as they say the rest is history. Apple is, IMO, due to become a $5 Trillion company. 
Microsoft's O/S was some rubbish that Ms Gates purloined from a guy in Seattle. It has always been sheit. Full of security holes, bugs you name it. MS ought to have dumped the whole kit and caboodle, and become what they have proven to be good at, Cloud. My Nutella's specialty. Except good old Windows has opened up a breach into the cloud and scuttled the works. 
I am no big fan of Android but it is heads and shoulders above Windows. 
Perhaps MS needs to team up with Google and license Android to run their PCs and Azure to manage the Cloud. 
Otherwise there will be days like these. 
 

Huh?!

dewme

Arggg!!!!!

edited July 2024

islander

I just had my CT scan canceled at a hospital In Massachusetts. The computer is down in the radiology department. Nurse blamed the "worldwide outage"

avon b7

Lettuce said:

avon b7 said:

My scheduled video conference with HP has hit the skids. I was told at 8am this morning that IT support were working on the problem but I doubt they'll apply the machine-by-machine workaround offered by Crowdstrike which requires booting into Safe Mode and deleting a file.

It's touch and go for the meeting which is scheduled to start in 30 minutes. 

It is more likely they'll wait for the fix from Crowdstrike to flush through the systems. 

Global automated updates should be rolled out in phases to catch these glitches before they become wildfires. 

I see some airlines still have boarding cards that can be handwritten for the lucky ones who have been able to get off the ground.

EDIT: meeting postponed.

The corrected update has already been rolled out. If the computer installed the wrong one it stopped working and the only way to fix it is to manually intervene. 

Yes. The issue was detected and corrected but this is what the CEO of Crowdstrike has said:


"Kurtz said “it could be some time for some systems” to return to normal, stressing that they would not “just automatically recover”." 

IT support at large corporations is being overwhelmed and while manual intervention would have allowed my meeting to go ahead, IT procedures don't allow for it. We couldn't switch to a tablet solution either because everything needs to be approved beforehand. Everything has to be done by the book. 

I have another video conference with a Japanese multinational which is still planned to go ahead later today. Fingers crossed.