Apple's recent software problems are bad, but shouldn't lead to knee-jerk personnel decisi...

Posted:
in General Discussion edited June 2018
By any account, Apple had a rough year with iOS 11 and macOS High Sierra from a software quality perspective -- but there are lessons to be learned and corrective measures to execute that can only be fully undertaken with the Apple leadership staying in place.




Editor's note: With WWDC upon us, rumors again suggest that Apple will slow the addition of new features, and instead focus on refining the operating system. The lessons learned in the end of 2017 and the beginning of 2018 are worth repeating.

In the span of a month, Apple was rocked by not just one software bug, but several. One of them was one of Apple's most severe security-related flaws yet -- the ability for a user to generate a Root account with the highest level of permissions possible, bypassing most of Apple's protections and security measures.

The fallout of the Root bug resulted in a series of crucial updates delivered through the App Store and automatically -- with their own foibles. Another issue developed on Dec. 2 with the iOS Notifications center, culminating in the what appears to be pre-emptive release of iOS 11.2 on early Saturday morning.

If Apple actually planned the release at that time (which seems unlikely) the Saturday morning release was certainly unparalleled in Apple's software release history.

The editorial and social media hue and cry for the virtual heads of Apple CEO Tim Cook or Senior Vice President Craig Federighi to be served up on a platter has begun. Should that happen, it will only make the situation worse.

Historical Parallel

Modern hardware is more complex than it has ever been. So many systems inter-relate and are so closely bound, that the user not being fully educated, a malfunctioning system, or software routine not operating well can have massive repercussions.

The nuclear-powered USS Thresher was the namesake of its class when it first put out to sea. It was the fastest, and quietest submarine in the world when it was built, and the most advanced weapons system of its time.




After a nine-month shipyard availability after initial sea trials to hammer out the bugs, the Thresher put out to sea. After a trip to test depth, the vessel was lost with all hands.

After recovery and reconstruction of the disaster, the Navy determined that the failure of a seawater piping system joint caused a cascading failure leading the the loss of the vessel. Simply, the joints in the piping were insufficient to the task, and quality assurance testing didn't spot the problem for a myriad of reasons.

The head of the U.S. Navy Nuclear Power program at the time was still the founder -- Admiral Hyman Rickover.

Vice Admiral Hyman G. Rickover with President John F. Kennedy
Vice Admiral Hyman G. Rickover with President John F. Kennedy


"I believe the loss of the Thresher should not be viewed solely as the result of failure of a specific braze, weld, system or component, but rather should be considered a consequence of the philosophy of design, construction and inspection that has been permitted in our naval shipbuilding programs," said Rickover. "I think it is important that we re-evaluate our present practices where, in the desire to make advancements, we may have forsaken the fundamentals of good engineering."

This saga of testing, and failure, may seem familiar to Apple fans.

The iPhone and a submarine?

The iPhone isn't a weapons platform, nor were any of the software bugs the cause of any loss of life. However, given modern life's reliance on the device, it can be used as a weapons platform against us.

An insecure Mac or iPhone could be used to surrender authentication methods or reset cloud access passwords. Properly attacked, in theory a bug like the no-password Root access could wipe out a user's entire stored data across iCloud or assorted Google data stores, using Apple's assorted lock and reset methods.

This isn't even including the potential damage from banking information going astray, or other financial information stolen from an attacked user.

The wake of a disaster

Admiral Rickover didn't lose his job because of the Thresher disaster, and it doesn't look like there were any mass-firings at the shipyard that did the maintenance at the time. Firing Admiral Rickover then would have set back Navy nuclear power, possibly never to recover. Instead, as a direct result of the maritime disaster, the U.S. Navy implemented the SUBSAFE quality assurance program. The program was a top to bottom renovation of the submarine supply chain, and parts accountability all the way from the assembly or manufacture of the part to installation.
"The devil is in the details, but so is salvation." - Vice Admiral Hyman G. Rickover.
Since then, the United States hasn't lost a vessel to a material failure. In the same time frame, several other nations have with the Russians having lost six -- but their parts and personnel vetting isn't as strict as the U.S. Navy.

Back to the original point -- Apple needs its own SUBSAFE system to protect its operating systems and with it, it's users, and it needs to start now.

Nothing worth doing is instant

Knee-jerk responses to large problems aren't good long-term solutions. SUBSAFE's basic premise was executed immediately, but didn't really get going for a few years. It took a long time to weed out bad parts from the supply chain and make other changes to the entire pipeline.

Tim Cook is a master of the supply chain, so that's not the problem. Cook was hand-selected by Steve Jobs, and was crafted by the Apple founder for many years to take the position.




In fact, the calls for Cook to step down for the crisis du jour are ridiculous, and any new selectee will not do as good a job with the supply chain. Additionally, any Apple head will take time to get up to speed in other matters given that there would not be any orderly turnover, compounding the problem.

Likewise, ditching Federighi solves nothing except a possible need for a scapegoat. The sudden void at the top will cause confusion, and a lack of focus in a company that needs to get its house in order regarding software quality assurance.

The human element

Regarding procedures and the operators, that process is constantly ongoing in the submarine fleet. With any luck, Apple will be able to take the time to do the same, and re-assess the situation internally with its in-house developers, and externally with users.

I'm not asking for a nine-month intensive classroom training phase followed up by closely supervised device operation before users get set loose, like the Navy demands of its engineers. But, Apple's security promises can only take uneducated users so far.

There will always be users whose device is considered an appliance. There will also always be AppleInsider readers who like to know why something works the way it does, and how to use the device to the maximum extent possible.

Ideally, the two will get together. The latter will talk to the former about security best practices, like physical security in conjunction with software security being the key components to ultimate user safety.

The path forward

Apple has promised changes. The company very quickly issued a statement about the Root bug after it was made public.

"We greatly regret this error and we apologize to all Mac users, both for releasing with this vulnerability and for the concern it has caused," wrote Apple. "Our customers deserve better. We are auditing our development processes to help prevent this from happening again."

This isn't about the march of Apple's version numbers causing problems, or any other related internet-founded silliness. Sierra's initial release 10.12.0 release could have easily been called El Capitan 10.11.7, and High Sierra's first version could have been called 10.12.6 -- but for marketing reasons, Apple incremented the version numbers and gave them fancy names. The same goes for iOS. The X=X+1 version number increment is more of a marketing tool than anything else.

An audit alone won't be enough to fix what ails the testing program, it appears. But, it is the first step on the road to recovery.

Extending the "life" of an operating system won't do anything, nor will lopping off the head of the company because of the misguided view that "this wouldn't have happened if Steve was alive."
«1345

Comments

  • Reply 1 of 85
    A very good article, grounded in history, reality and practice.

    Now come the rebuttals, grounded in naïveté, infantilism and solipsism.
    macseekermagman1979tmayjSnivelyrandominternetpersonlkruppandrewj5790russwMartin57watto_cobra
  • Reply 2 of 85
    wizard69wizard69 Posts: 13,377member
    Frankly the recent problems should surprise nobody.   For the last few years OS ipdates have been shipped to meet marketing event deadlines and as such quality has suffered significantly. 

    Frankly it is long past time for heads to roll at Apple but i dont think it is the software groups that have a problem.    Rather marketing needs to go.   More importantly Apple needs to focus on continous improvement to its products be they software or hardware.   It does little good to bundle a bunch of half tested features and ship them at a marketing party.   
    ecarlseenGG1jasenj1jongrallRealZoeSummersrogifan_newurahara
  • Reply 3 of 85
    Terrible analogies here. 

    Apple's software quality has been noticeably on the decline for many years now. They're well still ahead of their main competition (Google and Microsoft), but this may or may not last. It seems to have gotten worse around the time that they started doing public betas - I'm wondering if in-house testing was scaled down in conjunction with that.
    paisleydiscoanton zuykovhammeroftruthBubbaTwoAvieshekjasenj1
  • Reply 4 of 85
    GG1GG1 Posts: 483member
    Someone else pointed out in a previous article that Apple are still moving into the the spaceship, so that by itself is a big distraction.

    We may never know the real reason, but perhaps the move has also delayed the HomePod. And I've heard no update on the iMac Pro.

    Apple delayed the AirPod release, and it's now a bestseller. Be late and great!
    magman1979randominternetpersonpatchythepiratewatto_cobrajony0
  • Reply 5 of 85
    Mike WuertheleMike Wuerthele Posts: 6,861administrator
    ecarlseen said:
    Terrible analogies here. 

    Apple's software quality has been noticeably on the decline for many years now. They're well still ahead of their main competition (Google and Microsoft), but this may or may not last. It seems to have gotten worse around the time that they started doing public betas - I'm wondering if in-house testing was scaled down in conjunction with that.
    There have been show-stopping bugs in every version of System, MacOS, OS, MacOS X, and macOS, I've iterated most of them in another forum post.

    Having seen them all in one form or another, I don't think that it's gotten worse -- but I think the cacophony about it is louder as there are more users.
    StrangeDaysmagman1979jSnivelyrandominternetpersonlkruppwelshdogradarthekatrusswsmiffy31Martin57
  • Reply 6 of 85
    The idea of firing anyone over these issues is stupid. If there are consistent issues over a period of time then you can conclude there is a patterned approach to development which is insufficient, at which time you can let those responsible go.
    magman1979gregg thurmanjony0radarthekatrusswpatchythepiratewatto_cobrabadmonkDeadguy2322
  • Reply 7 of 85
    flaneurflaneur Posts: 4,526member
    ecarlseen said:
    Terrible analogies here. 

    Apple's software quality has been noticeably on the decline for many years now. They're well still ahead of their main competition (Google and Microsoft), but this may or may not last. It seems to have gotten worse around the time that they started doing public betas - I'm wondering if in-house testing was scaled down in conjunction with that.
    And I'm wondering if increasing complexity and the distraction of building and moving into a new headquarters might have anything to do with an increase in software "declne" — if that is even the case, objectively.
    watto_cobraSgt Storms(trooper)
  • Reply 8 of 85
    StrangeDaysStrangeDays Posts: 12,877member
    ecarlseen said:
    Terrible analogies here. 

    Apple's software quality has been noticeably on the decline for many years now. They're well still ahead of their main competition (Google and Microsoft), but this may or may not last. It seems to have gotten worse around the time that they started doing public betas - I'm wondering if in-house testing was scaled down in conjunction with that.
    Define noticeably? More defects, more defects relative to user base...? 

    As user Cyclonus wrote on another thread:

    I think it's pretty fair to say that there have been a number of equally "disastrous" situations in Apple software history with far more widespread and serious damage done to data and some cases hardware.  I'm looking at you 10.5 data loss bug.  Oh and the 10.6 guest user data loss bug.  Completely lose your data forever simply by copying and pasting or signing into a guest account.  How about iPhone OS 2 launch where simply launching the AIM app would totally brick an iPhone?  MobileMe iPocalypse anyone?  Don't even get me started with iTunes and it's track record for abominable horrors in it's time.  Oh how about Apple Music's disappearing libraries?  iTunes Music Match kidnapping songs?  I think you get it.  

    Apple software sees disastrous, embarrassing week with iOS springboard crash, macOS root u...
    randominternetpersonmacxpresskiltedgreenwatto_cobranetmagepropodjbishop1039talexyjony0
  • Reply 9 of 85
    Hi ya Mike;

    One of your better articles.  Yeah, the timetable for major updates needs to be longer.  I think a major update release could be 18 months in between releases.
    watto_cobraSgt Storms(trooper)
  • Reply 10 of 85
    RacerhomieXRacerhomieX Posts: 95unconfirmed, member
    I pray for Apple.
    nunzy
  • Reply 11 of 85
    glynhglynh Posts: 133member
    Good article and I agree with almost everything you say Mike.

    It's a pity however that 'scapegoat diplomacy' was in force when the time came to fire Scott Forstall as having listened to a. his dignified silence since this occurred and b. a recent interview with him in June 2017 where he comes across as full of Apple principles was allowed to happen.

    If anybody needs an example as to why calling for someone to be fired is counter-productive then his own experience seems to me to be a good example.

    I know he was supposed to be awkward, not easy to get on with and opinionated but then again wasn't Steve Jobs?
    cgWerkswatto_cobranetmagepreclarotipo
  • Reply 12 of 85
    At the very least they should both loose their 2017 bonus. 
  • Reply 13 of 85
    Glad someone (other than forum members in comments) is addressing the rote responses of those that say, no matter what the subject, that  “Tim, etal, must be fired” whenever a newsworthy article is published. I wonder sometimes if they’re using a keyboard shortcut for this. 

    As for “The X=X+1 version number increment is more of a marketing tool than anything else.” I’d have to disagree. Release management and source code maintenance is tightly tied to version numbering. Modern day development would be lost without it. 
    StrangeDayswatto_cobra
  • Reply 14 of 85
    k2kwk2kw Posts: 2,075member
    ecarlseen said:
    Terrible analogies here. 

    Apple's software quality has been noticeably on the decline for many years now. They're well still ahead of their main competition (Google and Microsoft), but this may or may not last. It seems to have gotten worse around the time that they started doing public betas - I'm wondering if in-house testing was scaled down in conjunction with that.
    There have been show-stopping bugs in every version of System, MacOS, OS, MacOS X, and macOS, I've iterated most of them in another forum post.

    Having seen them all in one form or another, I don't think that it's gotten worse -- but I think the cacophony about it is louder as there are more users.
    I wonder if the macOS group should be split off
    separate from iOS, tvOS, and watchOS.   IOS is the future.   MacOS is legacy past.

    Its been a big year for updates to iOS (iPad Pro and iPhone X changes), watchOS for LTE, and tvOS updates.   The homePod will need/have its own variation of iOS for the operating system (siriOS).   It's time to limit work on the mac .

    Other than that Cook needs to make sure there are enough developers.
  • Reply 15 of 85
    yojimbo007yojimbo007 Posts: 1,165member
    Whats wrong with issuing a software update on a saturday morning? ?
    russw
  • Reply 16 of 85
    wizard69 said:
    Frankly the recent problems should surprise nobody.   For the last few years OS ipdates have been shipped to meet marketing event deadlines and as such quality has suffered significantly. 

    Frankly it is long past time for heads to roll at Apple but i dont think it is the software groups that have a problem.    Rather marketing needs to go.   More importantly Apple needs to focus on continous improvement to its products be they software or hardware.   It does little good to bundle a bunch of half tested features and ship them at a marketing party.   
    I wonder if releasing major versions every 2 years like they used to would help.
  • Reply 17 of 85
    Rayz2016Rayz2016 Posts: 6,957member
    glynh said:
    Good article and I agree with almost everything you say Mike.

    It's a pity however that 'scapegoat diplomacy' was in force when the time came to fire Scott Forstall as having listened to a. his dignified silence since this occurred and b. a recent interview with him in June 2017 where he comes across as full of Apple principles was allowed to happen.

    If anybody needs an example as to why calling for someone to be fired is counter-productive then his own experience seems to me to be a good example.

    I know he was supposed to be awkward, not easy to get on with and opinionated but then again wasn't Steve Jobs?
    Scapegoat? I think he was fired because he wouldn’t take responsibility for the Apple maps along with the rest of Apple. Where’s the principle in that?
    StrangeDaysandrewj5790
  • Reply 18 of 85
    lkrupplkrupp Posts: 10,557member
    A very good article, grounded in history, reality and practice.

    Now come the rebuttals, grounded in naïveté, infantilism and solipsism.
    wizard69 said:
    Frankly the recent problems should surprise nobody.   For the last few years OS ipdates have been shipped to meet marketing event deadlines and as such quality has suffered significantly. 

    Frankly it is long past time for heads to roll at Apple but i dont think it is the software groups that have a problem.    Rather marketing needs to go.   More importantly Apple needs to focus on continous improvement to its products be they software or hardware.   It does little good to bundle a bunch of half tested features and ship them at a marketing party.   
    And here comes the naïveté and infantilism right on cue.
    edited December 2017 andrewj5790smiffy31watto_cobrajohntwolffnetmagepscooter63Deadguy2322
  • Reply 19 of 85
    A very good article, grounded in history, reality and practice.

    Now come the rebuttals, grounded in naïveté, infantilism and solipsism.
    I've been following Apple/AAPL since 1997 and investing via options since 2004.  In that period of time I have come to believe that easily 85% of so-called Apple followers are idiots of the first order.

    When it comes to investments I ignore any and all tales of woe attributed to Apple leadership, engineering, marketing, manufacture, demand, etc.  Since 2007 I've paid for everything I own (including home) with cash.

    It's easy to do when you ignore the idiots (chief among them today is Ming Chi Kuo).
    StrangeDaysradarthekatsmiffy31watto_cobraDeadguy2322
  • Reply 20 of 85
    zmaxman said:
    At the very least they should both loose their 2017 bonus. 
    That doesn't make any sense. Their bonus is based on performance. Performance is based on sales, which is based on satisfied customers, which is based on brand reputation. The existence of bugs, even a big one on the very minority platform that is the Mac, clearly hasn't reached a point where their customer base believes it's a problem. The Techie Echo Chamber of Doom, Worry, and Concern (tm) isn't the customer base.
    netmageanton zuykov
Sign In or Register to comment.