The kernel access issue is the key to this incident, and it's why Delta's CEO is right, even though he probably doesn't know why.
This should be an even bigger lesson as to why the EU's efforts to break into iOS are so profoundly wrong. Windows and MacOS are operating systems with deep legacy roots into the before times, when personal computers were standalone devices that weren't connected to a network. iOS was Apple re-thinking the whole paradigm, designed with constant network connectivity in mind. Constant internet connectivity is literally the (a?) reason for the i in iOS and iPhone. iOS was thus designed to protect the operating system entirely from outside access. This is why not only do you not need an antivirus program for your iPhone, you can't even buy one. Antivirus companies have whined about this for years, and have tried to imply that their lack of system access is somehow a bad idea.
To function, antivirus programs like Crowdstrike's have to exploit the very thing that they aim to protect against, which is outside access to the kernel, the core of the operating system. So in the recent case, a screw-up with their program essentially made it function exactly like a virus. When loaded onto Windows devices, it bricked them and lodged itself in the system, preventing a clean reboot. Because it wasn't a malicious attack (this time...) the fix was fairly easy, but required a lot of manual work to carry out.
MacOS had already cut off access to its kernel, and even further, has been edging closer to the iOS model, increasingly limiting installation of apps that haven't been pre-screened by Apple. You can still load them, but you have to agree that you're going around system protections to do it. Had MacOS been newly designed in 2007 as an always-connected system like iOS, it would have channeled all apps through the app store just like iOS. After the Crowdstrike debacle, Microsoft has indicated that it needs to follow Apple's lead and limit kernel access as well.
So even as Epic, Spotify and others are railing against the high walls around iOS, this experience should be showing people that iOS is actually the model to pursue, rather than loosening things up to go the other way in the misguided name of "freedom."
Ultimately what is probably needed is running virtual machines to serve up apps to basically dumb “terminals” at individual locations. Reduce the complexity of the work station. Reduce the risk of problems.
... the secondary cause is every Crowdstrike customer who blindly allowed the update to occur without doing their own testing and validation. Sure it would be nice to totally trust the vendor, but they don’t suffer the consequences. ...
There apparently was no mechanism for Crowdstrike customers to prevent the Rapid Response template update. This is an automated process that Crowdstrike intends to change in the future to allow customers granular control over the process (see Crowdstrike's Executive Summary):
What is CrowdStrike Doing to Prevent This From Happening Again?
... - Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed. ...
That’s fine, but the ultimate responsibility is Microsoft. What are they going to do, it’s their OS and they are at the top of the pyramid isn’t that why they get paid the big bucks?
Microsoft just has more homework to do, they have to fix AI Recall, get Windows to operate natively on Arm SOC'S or at least get Windows emulation to work on Arm (maybe they will get lucky on the fourth try), keep Qualcomm in line, hold Intels hand and last but not least cleanup the Kernel space by kicking out all those third-party companies.
I ask the Windows experts here: Does the move to Windows ARM allow MS to create a new locked-down OS like Apple has and once and for all fix many of the issues with their legacy problems on Intel?
The EU didn’t force Microsoft to do anything. The EU had complaints from security software vendors and asked Microsoft to respond. Microsoft responded with a solution a solution that provided security vendors the ability to live patch the Windows kernel. It was Microsoft’s solution and it was completely voluntary. That Microsoft is trying to blame the EU is just completely dishonest on their part.
That AppleInsider repeated Microsoft’s talking points without validating the claims was just sloppy. Paul Thurrott, a long time proponent of Windows and Microsoft, actually did work to validate the claim and called it for the BS it was. That a Microsoft fan put in more effort than AI is just embarrassing.
So, you are saying that if Microsoft would have responded with "No" or that "We are not going to change anything" that the EU would have allowed that? Did the EU sign off on the solution that Microsoft provided?
I ask the Windows experts here: Does the move to Windows ARM allow MS to create a new locked-down OS like Apple has and once and for all fix many of the issues with their legacy problems on Intel?
The ARM hardware doesn't change much, Microsoft can make a more locked down OS anytime they want but it would break some things. Windows is a big gaming platform and some of the biggest games use kernel-level anti-cheat software like Call of Duty. Any software that runs at the same level a user can access can be modified/hacked by the user.
It would be best if Microsoft eventually switched to a Unix system, this would fix a lot of problems right away and would attract more developers. If they can't secure a system at a higher level, they should build secure components into the OS and make APIs that 3rd parties can use.
Every major OS should be deployed read-only so that they remain bootable. System overrides can go into a writeable area and if the system crashes, it can disable the overrides.
Companies are letting far too much critical infrastructure depend on consumer-level systems. Delta says Crowdstrike cost them $0.5b, total costs are estimated to be over $5b. Critical systems that run airlines, hospitals, banks, ATMs etc should be bulletproof.
... the secondary cause is every Crowdstrike customer who blindly allowed the update to occur without doing their own testing and validation. Sure it would be nice to totally trust the vendor, but they don’t suffer the consequences. ...
There apparently was no mechanism for Crowdstrike customers to prevent the Rapid Response template update. This is an automated process that Crowdstrike intends to change in the future to allow customers granular control over the process (see Crowdstrike's Executive Summary):
What is CrowdStrike Doing to Prevent This From Happening Again?
... - Provide customers with greater control over the delivery of Rapid Response Content updates by allowing granular selection of when and where these updates are deployed. ...
That’s fine, but the ultimate responsibility is Microsoft. What are they going to do, it’s their OS and they are at the top of the pyramid isn’t that why they get paid the big bucks?
Microsoft just has more homework to do, they have to fix AI Recall, get Windows to operate natively on Arm SOC'S or at least get Windows emulation to work on Arm (maybe they will get lucky on the fourth try), keep Qualcomm in line, hold Intels hand and last but not least cleanup the Kernel space by kicking out all those third-party companies.
IMO, the ultimate responsibility is on Crowdstrike. They were the one who released the faulty update. From what I know, it's the first large-scale failure caused by a 3rd party security vendor since 2006, when MS step back from blocking kernel access because EC antitrust concerns. What MS did worked for many years, until now with the Crowdstrike bug.
Microsoft just has more homework to do, they have to fix AI Recall, get Windows to operate natively on Arm SOC'S or at least get Windows emulation to work on Arm (maybe they will get lucky on the fourth try), keep Qualcomm in line, hold Intels hand and last but not least cleanup the Kernel space by kicking out all those third-party companies.
The EU didn’t force Microsoft to do anything. The EU had complaints from security software vendors and asked Microsoft to respond. Microsoft responded with a solution a solution that provided security vendors the ability to live patch the Windows kernel. It was Microsoft’s solution and it was completely voluntary. That Microsoft is trying to blame the EU is just completely dishonest on their part.
That AppleInsider repeated Microsoft’s talking points without validating the claims was just sloppy. Paul Thurrott, a long time proponent of Windows and Microsoft, actually did work to validate the claim and called it for the BS it was. That a Microsoft fan put in more effort than AI is just embarrassing.
So, you are saying that if Microsoft would have responded with "No" or that "We are not going to change anything" that the EU would have allowed that? Did the EU sign off on the solution that Microsoft provided?
Here was the issue. For Windows Vista 64 bit Microsoft disallowed live patching of the kernel. Much like Apple has cut off kernel access in MacOS, iOS, iPadOS …. At the time 64bit Windows was a minor thing but was clear ly going to be the future of Windows computing. The idea was to use its small market share as the way to start transitions 3rd parties away from kernel access. Security companies had made a fortune with software that was able to live patch the kernel and while this didn’t present an immediate problem as the vast majority of Windows systems were all 32 bit and the product in question hadn’t shipped yet. It would mean that down the road they would have to rethink things. So, they went to the EU and complained. The EU asked Microsoft to respond to the complaint. Microsoft could have explained the importance of having to lock the kernel but they didn’t. They wanted to get the product out the door so they said they would offer limited patching of the kernel in an update. It was deliver in SP1. 3rd parties were satisfied and complaint was dropped.
Contrast that with what Microsoft is saying. Microsoft is saying the EU forced to provide the same level of kernel access that Microsoft has and that was why this situation occurred. The problem with that claim are the EU never made any design decisions nor did they require a change be made at all. Microsoft, fresh off a decade of antitrust litigation and wanting to ship a product decided to weaken it but they didn’t provide the same level of access, they gave limited access. So nothing MS is saying is true. The EU didn’t require them to give full access nor did they give full access.
What is worse, Microsoft knew that proving live patching access to third parties was a risk. That is why they were trying to remove it. The complaint to the EU happened 15 years ago. So over the last 15 years MS has had the opportunity to rectify the situation, lock down the kernel and ensure that its security software was playing on the same playing field as 3rd parties. Instead they opted to just continue with their half assed solution.
Crowedstike bares the majority of the responsibility here but Microsoft is in no way off the hook. The EU doesn’t bare any responsibility and they never prescribe architectural changes to Windows. Full stop.
Comments
This should be an even bigger lesson as to why the EU's efforts to break into iOS are so profoundly wrong. Windows and MacOS are operating systems with deep legacy roots into the before times, when personal computers were standalone devices that weren't connected to a network. iOS was Apple re-thinking the whole paradigm, designed with constant network connectivity in mind. Constant internet connectivity is literally the (a?) reason for the i in iOS and iPhone. iOS was thus designed to protect the operating system entirely from outside access. This is why not only do you not need an antivirus program for your iPhone, you can't even buy one. Antivirus companies have whined about this for years, and have tried to imply that their lack of system access is somehow a bad idea.
To function, antivirus programs like Crowdstrike's have to exploit the very thing that they aim to protect against, which is outside access to the kernel, the core of the operating system. So in the recent case, a screw-up with their program essentially made it function exactly like a virus. When loaded onto Windows devices, it bricked them and lodged itself in the system, preventing a clean reboot. Because it wasn't a malicious attack (this time...) the fix was fairly easy, but required a lot of manual work to carry out.
MacOS had already cut off access to its kernel, and even further, has been edging closer to the iOS model, increasingly limiting installation of apps that haven't been pre-screened by Apple. You can still load them, but you have to agree that you're going around system protections to do it. Had MacOS been newly designed in 2007 as an always-connected system like iOS, it would have channeled all apps through the app store just like iOS. After the Crowdstrike debacle, Microsoft has indicated that it needs to follow Apple's lead and limit kernel access as well.
So even as Epic, Spotify and others are railing against the high walls around iOS, this experience should be showing people that iOS is actually the model to pursue, rather than loosening things up to go the other way in the misguided name of "freedom."
/s
Microsoft just has more homework to do, they have to fix AI Recall, get Windows to operate natively on Arm SOC'S or at least get Windows emulation to work on Arm (maybe they will get lucky on the fourth try), keep Qualcomm in line, hold Intels hand and last but not least cleanup the Kernel space by kicking out all those third-party companies.
It would be best if Microsoft eventually switched to a Unix system, this would fix a lot of problems right away and would attract more developers. If they can't secure a system at a higher level, they should build secure components into the OS and make APIs that 3rd parties can use.
Every major OS should be deployed read-only so that they remain bootable. System overrides can go into a writeable area and if the system crashes, it can disable the overrides.
Companies are letting far too much critical infrastructure depend on consumer-level systems. Delta says Crowdstrike cost them $0.5b, total costs are estimated to be over $5b. Critical systems that run airlines, hospitals, banks, ATMs etc should be bulletproof.
Microsoft backs down over rivals' Vista access - Security Strategy - Breaking Business and Technology News at silicon.com (archive.org)
It looks like changes are coming. We'll how security companies will respond to those changes.
Windows resiliency: Best practices and the path forward - Microsoft Community Hub
Microsoft removed Recall many months ago
Microsoft’s all-knowing Recall AI feature is being delayed - The Verge
Windows have been native in ARM since the Surface Pro X in 2019. They also have native apps, like MS Office and VS Code.
Emulation also improved in the latest version of Windows with Prism.
How emulation works on Arm | Microsoft Learn
Qualcomm and Intel still working with MS.
We'll have to wait and see how they manage the kernel access. At least they are doing something to improve.