The irony is, we actually have a much more competitive market today than we did 25 years ago. Back in 1999, there was Wintel and not much else. Apple had about a 2% marketshare of the PC market, there was no smartphone market, almost all of the RISC guys were throwing in the towel out of fear of Intel, AMD was barely hanging on, etc etc.
Today, we have three major platform companies (Apple, Google, and Microsoft), not just one. We have real competition between Intel and AMD plus multiple very strong ARM-based competitors and RISC-V on the horizon.
This is basically a golden age of competition in computing platforms and the EC is trying to wreck it.
This seems to be totally a QA testing issue. Was any testing done?
That’s only part of it. The fact is, every developer will have some QA failures. Remember the launch of Apple Maps, the old FireWire deleting your hard drive on Mac OS update, etc? Too many ms and Google examples.
The issue here is not that a developer made a boo boo.
The issue is that a developer could make such a mistake and “gatekeepers@ who have safeguards and guardrails aren’t allowed to vet the update before it passes through and wrecks the planet.
This is entirely the result of irresponsible EU policy.
As I understand it, applying the update makes Windows stop working, resulting in the blue screen. This is not something hard for someone at CrowdStrike to test -- all they had to do is apply it to a test computer and reboot. Is it possible they released it without doing that?
It is possible they pushed the wrong version out of beta and the tested version is not what actually was applied.
Microsoft used to do that with their updates when I used to help support companies that had their server software. They regularly had to roll them back after “update tuesday”.
And now in the US we're letting judges and other power-hungry politicians replace tech- and science-savvy officers with themselves and their cohorts. To them, their beliefs (emotionally-induced decisions about what is true or false) are superior to factual knowledge (that which has been determined to be true or false). What could go wrong?
Pretty much sums up today’s political and lawmaking culture. Sad.
As I understand it, applying the update makes Windows stop working, resulting in the blue screen. This is not something hard for someone at CrowdStrike to test -- all they had to do is apply it to a test computer and reboot. Is it possible they released it without doing that?
I was involved with software testing for a number of years. The most common thing I think I heard that whole time was programmers whining, “the change does not impact function. There’s no need to test it”. So yes it is VERY possible. If the programmers thought they were only changing malware definitions or something they could have pushed to skip the tests.
The EC once again shows how clueless grandstanding politicians can cause havoc. The EC taking lefts and rights from Apple, Meta and now Microsoft. The only way to beat these ignorant folks is to band together and that’s what looks like is exactly happening. Nice!
It seems odd to come to AppleInsider and find people
repeating Microsoft PR talking points, but here we are. Strange times
indeed.
To understand what’s
going on, you need to go back to the 1990s, when Microsoft was
unconstrained by any thoughts of antitrust or monopoly law, and spent
quite a lot of time and effort decimating the third party software
industry. It had a massive advantage over the likes of Novell, Lotus,
WordPerfect and the rest because it could create and use APIs that
were private, while making second-rate APIs public.
It could also
deliberately break third party software with Windows updates. While
“DOS isn’t done till Lotus won’t run” was a myth, the truth
was that if an update broke a competitor’s app, that was good news
for Microsoft. And they definitely weren't going to go the extra mile to fix something that affected 1-2-3 if it didn't affect Excel.
Eventually this got
the attention of regulators including the DoJ and EC (European
Commission). It’s EXACTLY the kind of behaviour that you are
absolutely not allowed to do under antitrust law. When you have a
dominant position – and no one doubted, or doubts, MS has this in
operating systems – you just can’t get away with it.
In 2004, the EC case
was pretty-much over. Microsoft agreed it had been bad, and offered
to publish its APIs, and apply a level playing field – which meant
its own applications weren’t allowed to use special
“Microsoft-only” APIs. Anything Microsoft’s apps could do HAD
to be available to others.
Did Microsoft stick
to this agreement like a good little boy? Of course it didn’t.
So in 2006, a group
of software companies complained it to the EC, through a coalition
called the European Committee for Interoperable Standards (ECIS).
Despite the name, ECIS was largely US companies, including IBM,
Adobe, Oracle, and McAfee. By 2007, the EC had investigated and found
that yes, Microsoft had failed to live up to its agreements. It got
fined, and the EC asked Microsoft to propose new, specific remedies
to make sure it didn’t happen again.
In 2009, those
agreements were signed. And in them, there is a specific part –
section C (42) – which deals with security software. Now you might
get the impression from what Microsoft is saying now, something
that’s being repeated by people who can’t be bothered to look up
agreements AKA “pundits”, that this mandates kernel level access
for third parties.
Reader, it does
nothing of the sort. It simply states that Microsoft has to make
available – and document – whatever APIs its own software uses.
The company could do what
Apple has done and move access for EDR (endpoint
detection and response) software out of the kernel. It has chosen not
to do this.
So
no, Microsoft hasn’t been “ordered” by the big bad EU to do
anything other than stop its old tricks of giving its own
applications advantages that no third party could ever have. It
hasn’t moved EDR out of the kernel because, at least back in 2009,
the Windows kernel was a mess and developing equivalent APIs was
going to be expensive.
Do
I blame Microsoft? Not really: Windows is what it is, and keeping it
secure is hard. I don’t believe its the platform vendors fault if,
using legitimate methods, a third party messes up a patch. That’s
entirely down to Crowdstrike.
But
is it the EC’s fault? Absolutely not. Stopping companies like
Microsoft from destroying competition not by better products but by
leveraging ownership of a platform is exactly the thing antitrust
bodies are set up to do. It’s what the DoJ did to IBM in 1956, and
without that judgement we would all be still using mainframes from
Big Blue.
Yep. The geniuses at the EU made it so cloudstrike can bypass Microsoft safeguards entirely and push their own junk directly onto Microsoft systems.
Sigh. No, no they really didn't do that. Please go and look at this web page.
From there, you can download the Word document called "Microsoft Interoperability Undertaking", which is the agreement that Microsoft is claiming "made" it give kernel access to third parties.
Go to Section C, paragraph 42, which is the undertaking concerning security software. Find the bit which mandates kernel access. I'll wait. And I'll be waiting a long time, because no such mandate exists, despte what certain pundits with bees in their bonnet about the EU might have told you.
This is basically a golden age of competition in computing platforms and the EC is trying to wreck it.
It's a golden age of competition in computing because the DoJ and EU both took action to rein in Microsoft in the late 90s and early 2000s. Maybe you're not old enough to remember when Microsoft could get away with anything he wanted, but trust me, it wasn't fun. Ask Novell. Or WordPerfect. Or Lotus. All of whom had better products which got steamrollered because Microsoft controlled Windows.
This seems to be totally a QA testing issue. Was any testing done?
So the answer to this is, "it's complicated". Some of this is going to be a bit simplified, but it's accurate enough.
Software on Windows can run in two modes: kernel mode; and user mode. User mode software shouldn't ever be able to cause a BSOD.
Security software needs to run in kernel mode. There are good reasons for this: malware often hides deep in the OS in places where user mode software can't find it. CrowdStrike Falcon works like a device drive, which allows it to reside in kernel mode and access system data structures and services.
"Heck," you're thinking, "so can anyone write a device driver and get their software running in kernel mode?" Well, no: Windows will display a warning unless a driver has passed Microsoft's WHQL testing process. In some cases, Windows won't even allow the driver to run.
Falcon is WHQL certified, so it *should* be pretty robust and not cause a BSOD. But there's a catch: it relies on dynamic definition files, which are deployed to update its configuration. From what I hear, Crowdstrike accidentally deployed one which contained nothing but zeros, which led to a catastrophic error. In other words, they simply deployed the wrong file. No testing will catch that -- it's a file that wasn't meant to be deployed at all.
The problem last week had nothing to do with the EU. It was sloppy coding, sloppy testing and with little to no resilience built into the whole process.
You keep living that fantasy, but the facts are that EU politicians with no appreciable technical knowledge are making decisions that shape technology with no idea of the possible side-effects of those decisions, nor any concern other than that they see their decisions financially benefitting EU companies at the expense of American companies.
This is the equivalent at a company of HR driving tech decisions and it's working out about as well as can be expected. The end result will be lower quality, less secure , less resilient hardware and software. We will see many more incidents like this in the future that all lead back to the same place: EU politicians.
This is basically a golden age of competition in computing platforms and the EC is trying to wreck it.
It's a golden age of competition in computing because the DoJ and EU both took action to rein in Microsoft in the late 90s and early 2000s. Maybe you're not old enough to remember when Microsoft could get away with anything he wanted, but trust me, it wasn't fun. Ask Novell. Or WordPerfect. Or Lotus. All of whom had better products which got steamrollered because Microsoft controlled Windows.
I am old enough, which is why I made the point -- I remember it very well.
It's possible that past anti-trust action -- or the threat of such action -- contributed to the current landscape. I can see an argument that the fear of anti-trust action led MS to continue to support Office on the Mac in the late 90s to early 2000s. If MS had withdrawn Office support, it might have ended the Mac. So that's a potential point in favor of DOJ.
But so what? The fact that some anti-trust actions taken by the government are smart and well implemented doesn't mean all actions are smart and well implemented. I think the EC has gone off the rails and is diving into a level of centrally planned micromanaging that is counterproductive. In the US, there are zealots on the left who are inclined to do the same thing, although in general it's anti-government zealots on the right that are the bigger problem in the US.
This is basically a golden age of competition in computing platforms and the EC is trying to wreck it.
It's a golden age of competition in computing because the DoJ and EU both took action to rein in Microsoft in the late 90s and early 2000s. Maybe you're not old enough to remember when Microsoft could get away with anything he wanted, but trust me, it wasn't fun. Ask Novell. Or WordPerfect. Or Lotus. All of whom had better products which got steamrollered because Microsoft controlled Windows.
I am old enough, which is why I made the point -- I remember it very well.
It's possible that past anti-trust action -- or the threat of such action -- contributed to the current landscape. I can see an argument that the fear of anti-trust action led MS to continue to support Office on the Mac in the late 90s to early 2000s. If MS had withdrawn Office support, it might have ended the Mac. So that's a potential point in favor of DOJ.
But so what? The fact that some anti-trust actions taken by the government are smart and well implemented doesn't mean all actions are smart and well implemented. I think the EC has gone off the rails and is diving into a level of centrally planned micromanaging that is counterproductive. In the US, there are zealots on the left who are inclined to do the same thing, although in general it's anti-government zealots on the right that are the bigger problem in the US.
There's absolutely no doubt at all that Microsoft in the late 90s and early 2000s was constrained by antitrust, and not just because of the specific rules. Like all companies subject to antitrust action, it will have pre-emptively ruled out certain activities which, in the past, it would happily have done. Bill Gates has talked about how antitrust limited what they could do in mobile, for example. And yet there are still commentators out there who insist antitrust action had no effect! (Ben Thompson, if you're reading this, I am looking at you).
As for micro-managing, again, I would encourage you to go look at the actual 2009 agreement, which is about as far from technical micro-managing as it's possible to get. There are *no* technical requirements in it, just requirements that whatever APIs Microsoft's own apps use, they have to allow others to do too. If anything, it's *less* onerous than the 1956 IBM consent decree which basically created the PC market, and which made IBM publish not only software manuals but schematics too.
The EC once again shows how clueless grandstanding politicians can cause havoc. The EC taking lefts and rights from Apple, Meta and now Microsoft. The only way to beat these ignorant folks is to band together and that’s what looks like is exactly happening. Nice!
It seems odd to come to AppleInsider and find people
repeating Microsoft PR talking points, but here we are. Strange times
indeed.
To understand what’s
going on, you need to go back to the 1990s, when Microsoft was
unconstrained by any thoughts of antitrust or monopoly law, and spent
quite a lot of time and effort decimating the third party software
industry. It had a massive advantage over the likes of Novell, Lotus,
WordPerfect and the rest because it could create and use APIs that
were private, while making second-rate APIs public.
It could also
deliberately break third party software with Windows updates. While
“DOS isn’t done till Lotus won’t run” was a myth, the truth
was that if an update broke a competitor’s app, that was good news
for Microsoft. And they definitely weren't going to go the extra mile to fix something that affected 1-2-3 if it didn't affect Excel.
Eventually this got
the attention of regulators including the DoJ and EC (European
Commission). It’s EXACTLY the kind of behaviour that you are
absolutely not allowed to do under antitrust law. When you have a
dominant position – and no one doubted, or doubts, MS has this in
operating systems – you just can’t get away with it.
In 2004, the EC case
was pretty-much over. Microsoft agreed it had been bad, and offered
to publish its APIs, and apply a level playing field – which meant
its own applications weren’t allowed to use special
“Microsoft-only” APIs. Anything Microsoft’s apps could do HAD
to be available to others.
Did Microsoft stick
to this agreement like a good little boy? Of course it didn’t.
So in 2006, a group
of software companies complained it to the EC, through a coalition
called the European Committee for Interoperable Standards (ECIS).
Despite the name, ECIS was largely US companies, including IBM,
Adobe, Oracle, and McAfee. By 2007, the EC had investigated and found
that yes, Microsoft had failed to live up to its agreements. It got
fined, and the EC asked Microsoft to propose new, specific remedies
to make sure it didn’t happen again.
In 2009, those
agreements were signed. And in them, there is a specific part –
section C (42) – which deals with security software. Now you might
get the impression from what Microsoft is saying now, something
that’s being repeated by people who can’t be bothered to look up
agreements AKA “pundits”, that this mandates kernel level access
for third parties.
Reader, it does
nothing of the sort. It simply states that Microsoft has to make
available – and document – whatever APIs its own software uses.
The company could do what
Apple has done and move access for EDR (endpoint
detection and response) software out of the kernel. It has chosen not
to do this.
So
no, Microsoft hasn’t been “ordered” by the big bad EU to do
anything other than stop its old tricks of giving its own
applications advantages that no third party could ever have. It
hasn’t moved EDR out of the kernel because, at least back in 2009,
the Windows kernel was a mess and developing equivalent APIs was
going to be expensive.
Do
I blame Microsoft? Not really: Windows is what it is, and keeping it
secure is hard. I don’t believe its the platform vendors fault if,
using legitimate methods, a third party messes up a patch. That’s
entirely down to Crowdstrike.
But
is it the EC’s fault? Absolutely not. Stopping companies like
Microsoft from destroying competition not by better products but by
leveraging ownership of a platform is exactly the thing antitrust
bodies are set up to do. It’s what the DoJ did to IBM in 1956, and
without that judgement we would all be still using mainframes from
Big Blue.
OMG! Someone who knows what they’re talking about instead of the uninformed knee-jerk anti-EU drivel purveyors who have read too much online “comment”. Thank you Ian.
This seems to be totally a QA testing issue. Was any testing done?
my understanding is testing probably was done but the file got corrupt somehow when deployed it was full of nulls when crashed the kernel with null pointer exceptions. And there was no mechanism for them to track clients were not coming back online after they updated and pull it before it got out of control.
This seems to be totally a QA testing issue. Was any testing done?
So the answer to this is, "it's complicated". Some of this is going to be a bit simplified, but it's accurate enough.
Software on Windows can run in two modes: kernel mode; and user mode. User mode software shouldn't ever be able to cause a BSOD.
Security software needs to run in kernel mode. There are good reasons for this: malware often hides deep in the OS in places where user mode software can't find it. CrowdStrike Falcon works like a device drive, which allows it to reside in kernel mode and access system data structures and services.
"Heck," you're thinking, "so can anyone write a device driver and get their software running in kernel mode?" Well, no: Windows will display a warning unless a driver has passed Microsoft's WHQL testing process. In some cases, Windows won't even allow the driver to run.
Falcon is WHQL certified, so it *should* be pretty robust and not cause a BSOD. But there's a catch: it relies on dynamic definition files, which are deployed to update its configuration. From what I hear, Crowdstrike accidentally deployed one which contained nothing but zeros, which led to a catastrophic error. In other words, they simply deployed the wrong file. No testing will catch that -- it's a file that wasn't meant to be deployed at all.
This seems to be totally a QA testing issue. Was any testing done?
So the answer to this is, "it's complicated". Some of this is going to be a bit simplified, but it's accurate enough.
Software on Windows can run in two modes: kernel mode; and user mode. User mode software shouldn't ever be able to cause a BSOD.
Security software needs to run in kernel mode. There are good reasons for this: malware often hides deep in the OS in places where user mode software can't find it. CrowdStrike Falcon works like a device drive, which allows it to reside in kernel mode and access system data structures and services.
"Heck," you're thinking, "so can anyone write a device driver and get their software running in kernel mode?" Well, no: Windows will display a warning unless a driver has passed Microsoft's WHQL testing process. In some cases, Windows won't even allow the driver to run.
Falcon is WHQL certified, so it *should* be pretty robust and not cause a BSOD. But there's a catch: it relies on dynamic definition files, which are deployed to update its configuration. From what I hear, Crowdstrike accidentally deployed one which contained nothing but zeros, which led to a catastrophic error. In other words, they simply deployed the wrong file. No testing will catch that -- it's a file that wasn't meant to be deployed at all.
Well written. I do take exception to one thing though.
<blockquote>In other words, they simply deployed the wrong file. No testing will catch that -- it's a file that wasn't meant to be deployed at all. <blockquote>
As some who has written installation packages, with any company worth their salt, the install package goes through testing, also. That would have caught the zero file.
Comments
Well said.
That’s only part of it. The fact is, every developer will have some QA failures. Remember the launch of Apple Maps, the old FireWire deleting your hard drive on Mac OS update, etc? Too many ms and Google examples.
Pretty much sums up today’s political and lawmaking culture. Sad.
To understand what’s going on, you need to go back to the 1990s, when Microsoft was unconstrained by any thoughts of antitrust or monopoly law, and spent quite a lot of time and effort decimating the third party software industry. It had a massive advantage over the likes of Novell, Lotus, WordPerfect and the rest because it could create and use APIs that were private, while making second-rate APIs public.
It could also deliberately break third party software with Windows updates. While “DOS isn’t done till Lotus won’t run” was a myth, the truth was that if an update broke a competitor’s app, that was good news for Microsoft. And they definitely weren't going to go the extra mile to fix something that affected 1-2-3 if it didn't affect Excel.
Eventually this got the attention of regulators including the DoJ and EC (European Commission). It’s EXACTLY the kind of behaviour that you are absolutely not allowed to do under antitrust law. When you have a dominant position – and no one doubted, or doubts, MS has this in operating systems – you just can’t get away with it.
In 2004, the EC case was pretty-much over. Microsoft agreed it had been bad, and offered to publish its APIs, and apply a level playing field – which meant its own applications weren’t allowed to use special “Microsoft-only” APIs. Anything Microsoft’s apps could do HAD to be available to others.
Did Microsoft stick to this agreement like a good little boy? Of course it didn’t.
So in 2006, a group of software companies complained it to the EC, through a coalition called the European Committee for Interoperable Standards (ECIS). Despite the name, ECIS was largely US companies, including IBM, Adobe, Oracle, and McAfee. By 2007, the EC had investigated and found that yes, Microsoft had failed to live up to its agreements. It got fined, and the EC asked Microsoft to propose new, specific remedies to make sure it didn’t happen again.
In 2009, those agreements were signed. And in them, there is a specific part – section C (42) – which deals with security software. Now you might get the impression from what Microsoft is saying now, something that’s being repeated by people who can’t be bothered to look up agreements AKA “pundits”, that this mandates kernel level access for third parties.
Reader, it does nothing of the sort. It simply states that Microsoft has to make available – and document – whatever APIs its own software uses. The company could do what Apple has done and move access for EDR (endpoint detection and response) software out of the kernel. It has chosen not to do this.
So no, Microsoft hasn’t been “ordered” by the big bad EU to do anything other than stop its old tricks of giving its own applications advantages that no third party could ever have. It hasn’t moved EDR out of the kernel because, at least back in 2009, the Windows kernel was a mess and developing equivalent APIs was going to be expensive.
Do I blame Microsoft? Not really: Windows is what it is, and keeping it secure is hard. I don’t believe its the platform vendors fault if, using legitimate methods, a third party messes up a patch. That’s entirely down to Crowdstrike.
But is it the EC’s fault? Absolutely not. Stopping companies like Microsoft from destroying competition not by better products but by leveraging ownership of a platform is exactly the thing antitrust bodies are set up to do. It’s what the DoJ did to IBM in 1956, and without that judgement we would all be still using mainframes from Big Blue.
Yes it does. And CrowdStrike Falcon has been linked to kernel panics on Linux in the past. This one didn't affect Linux though.
It's a golden age of competition in computing because the DoJ and EU both took action to rein in Microsoft in the late 90s and early 2000s. Maybe you're not old enough to remember when Microsoft could get away with anything he wanted, but trust me, it wasn't fun. Ask Novell. Or WordPerfect. Or Lotus. All of whom had better products which got steamrollered because Microsoft controlled Windows.
Falcon is WHQL certified, so it *should* be pretty robust and not cause a BSOD. But there's a catch: it relies on dynamic definition files, which are deployed to update its configuration. From what I hear, Crowdstrike accidentally deployed one which contained nothing but zeros, which led to a catastrophic error. In other words, they simply deployed the wrong file. No testing will catch that -- it's a file that wasn't meant to be deployed at all.
This is the equivalent at a company of HR driving tech decisions and it's working out about as well as can be expected. The end result will be lower quality, less secure , less resilient hardware and software. We will see many more incidents like this in the future that all lead back to the same place: EU politicians.
This *could* have affected Linux, too. We really got lucky that it didn't, given the amount of infrastructure that runs on it.
A. giant meteor
B. world war III
C. global warming
D. bad Windows security update ✓
It's possible that past anti-trust action -- or the threat of such action -- contributed to the current landscape. I can see an argument that the fear of anti-trust action led MS to continue to support Office on the Mac in the late 90s to early 2000s. If MS had withdrawn Office support, it might have ended the Mac. So that's a potential point in favor of DOJ.
But so what? The fact that some anti-trust actions taken by the government are smart and well implemented doesn't mean all actions are smart and well implemented. I think the EC has gone off the rails and is diving into a level of centrally planned micromanaging that is counterproductive. In the US, there are zealots on the left who are inclined to do the same thing, although in general it's anti-government zealots on the right that are the bigger problem in the US.
<blockquote>In other words, they simply deployed the wrong file. No testing will catch that -- it's a file that wasn't meant to be deployed at all. <blockquote>
As some who has written installation packages, with any company worth their salt, the install package goes through testing, also. That would have caught the zero file.