Consumer Reports now recommends MacBook Pro after Apple software fix

Posted:
in Current Mac Hardware edited January 2017
In a bit of backpedaling, Consumer Reports on Thursday said it can now recommend Apple's new MacBook Pro lineup after a software fix remedied battery life inconsistencies found in previous testing.




Consumer Reports in an update to its original MacBook Pro evaluation said it has finished retesting the battery life of MacBook Pro laptops running a macOS 10.12.3 beta released earlier this week. As the battery problems are no longer present, the publication can now recommend the laptop series.

"With the updated software, the three MacBook Pros in our labs all performed well, with one model running 18.75 hours on a charge. We tested each model multiple times using the new software, following the same protocol we apply to hundreds of laptops every year," the report said.

The reevaluation comes after the late-2016 MacBook Pro lineup failed to garner the publication's recommended rating for the first time ever. Specifically, Consumer Reports blamed the rating on extreme variations in battery runtime tests.

The publication found wild fluctuations in battery life runtimes for each of Apple's new MacBook Pro models, both with and without Touch Bar. In some cases operating times vacillated from 16 hours in one test to as little as 3.75 hours in another.

Following the poor showing, Apple revealed it was working with the publication to resolve what at the time appeared to be unusual results not indicative of real-world use. Apple spends substantial capital on special machinery, facilities and man hours to perform rigorous quality assurance testing before putting a product up for sale, suggesting the Consumer Reports test was somehow anomalous.





Earlier this week, Apple in a statement to AppleInsider said a hidden developer setting in its Safari web browser was enabled by Consumer Reports during testing and triggered an obscure bug that led to inconsistent battery life readings. The issue was resolved in a macOS 10.12.3 beta that should see public release soon.

"This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab," Apple said. "After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life."
«13

Comments

  • Reply 1 of 56
    dws-2dws-2 Posts: 148member
    The real question is who could recommend Consumer Reports?

    I love the idea of an independent body that fairly evaluates products, but their methods lead to wrong results as often as right ones. I have a 15" MBP with Touch Bar, and I can tell you there is absolutely no way it ever lasted 19 hours web browsing with the screen on. It lasts me a little more than what Apple says if I'm web browsing and about 5-6 hours if I'm doing work.
    edited January 2017 pulseimagesdjkfisherNaiyas
  • Reply 2 of 56
    No. Not after "software fix", after their FRAUD was exposed to everyone.

    Running the machine with a hidden developer setting that is DESIGNED to reduce performance, and then not disclosing that fact, and then claiming an independant test shows the machine under-performs the manufacturers claims is FRAUD.  Possibly Libel/Slander as well.

    CU is not a credible organization.
    Mikeymikepulseimageslostkiwidjkfisherhlee1169monstrositytallest skilwatto_cobra
  • Reply 3 of 56
    How is this a backpedal? CR had an issue with the battery and reported it. The report got Apple's attention and a fix. As the problem doesn't exist any longer they change their recommendation.
    edited January 2017 jbdragonpulseimagesfarjamedyojimbo007levimike1williamlondonwaverboy
  • Reply 4 of 56
    I cannot agree more with the consensus that this was not a representative way to measure battery performance.  But it sounds like consumer reports has done this for ALL laptops, disabling the browser cache, not as a way to measure real world battery use during browsing, but as a way to use up the battery in a comparable way across all systems.  

    So while I agree that default browser settings on each platform make sense, then it also is only fair for CR to go back and test all the OTHER laptops with their browser caches enabled - then all of them will also show much longer battery life.

    The bottom line is Apple was NOT singled out in approach, so the real fix is to change it for everyone.

    pulseimagesfarjamedwilliamlondon
  • Reply 5 of 56
    No. Not after "software fix", after their FRAUD was exposed to everyone.

    Running the machine with a hidden developer setting that is DESIGNED to reduce performance, and then not disclosing that fact, and then claiming an independant test shows the machine under-performs the manufacturers claims is FRAUD.  Possibly Libel/Slander as well.

    CU is not a credible organization.
    You mean CR? As the setting was due to an actual bug, no matter how rare, I doubt there will be a libel or slander suit. Glad that Apple fixed it. Now if they could get more RAM in there I'd be happy. Maybe next year. 
    edited January 2017 pulseimages
  • Reply 6 of 56
    lkrupplkrupp Posts: 5,632member
    The kerfuffle uncovered a bug. Apple admits a bug caused the issue for CR’s testing. The bug was squashed. The new tests caused CR to change its recommendation. We should be happy.
    pulseimagesanantksundaramnocreatorblueretrogustowilliamlondonteaearlegreyhotwaverboywatto_cobrajony0
  • Reply 7 of 56
    jungmarkjungmark Posts: 6,482member
    Could've ask Apple to review when they got very inconsistent results. Could've saved us from this mess. 
    StrangeDayspulseimagesration alretrogustomonstrosity
  • Reply 8 of 56
    CR publishing the non-recommendation was silly because the scenario they were tested in is not an accurate representation of normal use. the bug only hit the battery in such a dramatic fashion due to the disabled cache and atypical repetition/automation. the batteries weren't defective. so because it was the test scenario, and obviously so, the responsible thing to do would have been to work with Apple first, before publishing, not after. instead, by publishing a non-recommendation, they suggested the notebooks had defective batteries. poor journalism. 
    caliwilliamlondonNaiyaschasm
  • Reply 9 of 56
    CR publishing the non-recommendation was silly because the scenario they were tested in is not an accurate representation of normal use. the bug only hit the battery in such a dramatic fashion due to the disabled cache and atypical repetition/automation. the batteries weren't defective. so because it was the test scenario, and obviously so, the responsible thing to do would have been to work with Apple first, before publishing, not after. instead, by publishing a non-recommendation, they suggested the notebooks had defective batteries. poor journalism. 
    Agree. And especially given that it was a 'browser' related (Safari) issue.
    edited January 2017 caliwilliamlondon
  • Reply 10 of 56
    This doesn't mean CR did their test incorrectly, or unfairly.  They were not the only reviewers seeing disappointing battery life and then of course you have the real world complaints from actual customers.  I hope in the coming weeks we see that this bug fix solves the battery complaints. 
    anantksundaramwilliamlondonteaearlegreyhot
  • Reply 11 of 56
    jungmark said:
    Could've ask Apple to review when they got very inconsistent results. Could've saved us from this mess. 
    There was no mess. It was simply and correctly resolved in a couple of weeks. 

    It was was no big deal. 
    hmmwilliamlondonteaearlegreyhotwaverboy
  • Reply 12 of 56
    SoliSoli Posts: 6,141member
    jungmark said:
    Could've ask Apple to review when they got very inconsistent results. Could've saved us from this mess. 
    There was no mess. It was simply and correctly resolved in a couple of weeks. 

    It was was no big deal. 
    I disagree. You don't get 4.5 to 19.5 hours running the same test and then publish results that affect buying decisions for the company with the largest mindshare. This was an opportunity for CR to get extra press. I wouldn't call their actions nefarious, but I would say that their methodology with CE has repeatedly shown to be shoddy. If you want sold reviews of soft-coded technology then use Wirecutter.
    ration alpulseimages
  • Reply 13 of 56
    tzeshantzeshan Posts: 1,589member
    Consumer may have committed a user error in testing.  It is still a software bug because Apple can fix the problem. A software follows a flow chart. R&D arguing that the user should not lead the software go to a particular mode is simply not responsible. 
    teaearlegreyhot
  • Reply 14 of 56
    NY1822NY1822 Posts: 477member
    Apple did not "fix" any bug....the only evidence we have of what Apple did is in the last line of the article:
    "After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life"

    Consumer reports comment of "with the updated software" could mean "us doing it the correct way this time" NOT "Apple updated their software"
    edited January 2017 Rayz2016tallest skil
  • Reply 15 of 56
    nhtnht Posts: 4,005member
    Soli said:
    jungmark said:
    Could've ask Apple to review when they got very inconsistent results. Could've saved us from this mess. 
    There was no mess. It was simply and correctly resolved in a couple of weeks. 

    It was was no big deal. 
    I disagree. You don't get 4.5 to 19.5 hours running the same test and then publish results that affect buying decisions for the company with the largest mindshare. This was an opportunity for CR to get extra press. I wouldn't call their actions nefarious, but I would say that their methodology with CE has repeatedly shown to be shoddy. If you want sold reviews of soft-coded technology then use Wirecutter.
    They didn't even follow the same procedure as all other tests on other laptops.  The procedure in all other tests was to average the results and compute the score. Instead they decided that because their test was broken to only use the lowest score.


    williamlondonRayz2016
  • Reply 16 of 56
    What I read here was that CR found a bug in Apple's developer software, Apple fixed the developer software and CR now can recommend MBP for average consumer (who are not developer) to use. Can someone tell me if I were the only one who see something really strange about this?
    pulseimageswilliamlondon
  • Reply 17 of 56
    linkmanlinkman Posts: 720member
    Consumer Reports still doesn't come up with some real-world figures again. They report getting 15.75 hours, 18.75 hours, and 17.25 hours. Apple states up to 10 hours wireless web and up to 10 hours iTunes movie playback. Yes, CR should obtain very consistent figures with their testing method but it's only testing web browsing.
    pulseimages
  • Reply 18 of 56
    jungmark said:
    Could've ask Apple to review when they got very inconsistent results. Could've saved us from this mess. 
    That requires common sense. I mean, don't consumers enable hidden developer mode features and perform scripted battery rundown tests all the time?
    williamlondon
  • Reply 19 of 56
    anomeanome Posts: 867member
    This isn't doing Consumer Report's reputation any good. They're basically admitting they really didn't know what they were doing in these tests. You'd think a respectable testing organisation would have gone to Apple and said "We're seeing something funny here, any ideas?" rather than jumping straight to press with "FAIL!"
    kevin kee said:
    What I read here was that CR found a bug in Apple's developer software, Apple fixed the developer software and CR now can recommend MBP for average consumer (who are not developer) to use. Can someone tell me if I were the only one who see something really strange about this?

    It is a bit strange, and has been covered on other threads. CR thought they were simulating a "real world situation" by using a developer setting. Actually testing in "real world conditions" might have gotten them a different result, but that's harder to automate.

  • Reply 20 of 56
    How about mentioning that baseline apple software of late seems buggier then ever. Do they even test the non standard modes at apple at all? Disabling cache is not that uncommon, I think some blame at apple's software quality control is in order - pony up 500 million and hire more testers, it's a rounding error in a stock buyback for them. 
    williamlondon
Sign In or Register to comment.