Apple says hidden Safari setting led to flawed Consumer Reports MacBook Pro battery tests

Posted:
in Current Mac Hardware edited January 2017
Battery tests conducted by Consumer Reports showing inconsistent uptime with Apple's new MacBook Pro were inaccurate, Apple said on Tuesday, explaining that the publication had enabled a hidden setting in its Safari browser.




In a statement issued to AppleInsider, Apple said it worked with Consumer Reports over the holidays, and determined that its testing methods for the new MacBook Pro were flawed. The company had initially announced just before Christmas that it was cooperating with Consumer Reports to determine what went wrong with the MacBook Pro units they tested.
Apple says battery inconsistency was caused by a hidden Safari setting for developers which turns off the browser cache. The setting is not used by customers.
"We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache," Apple's statement reads. "This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab.

"After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life. We have also fixed the bug uncovered in this test. This is the best pro notebook we've ever made, we respect Consumer Reports and we're glad they decided to revisit their findings on the MacBook Pro."

The Safari fix is available to beta testers, presumably in the latest pre-release build of macOS 10.12.3, made available to registered developers on Tuesday. Apple says the full fix will be provided to the public in an upcoming macOS update.

Previous testing by the publication led Consumer Reports to not recommend any new MacBook Pro model due to battery life concerns. Their in-house testing revealed wild fluctuations in unplugged operating survivability, in some cases ranging from 16 hours to as little as 3.75 hours.

In its initial findings, the publication left the door open for an Apple response, noting the battery life of many modern products are "influenced" by software updates.

"If Apple updates its software in a way that the company claims will substantively change battery performance, we will conduct fresh tests," the original report said.
«13456

Comments

  • Reply 1 of 118
    kenckenc Posts: 195member
    So CR turned off the browser cache...d'oh!
    bradmacproredgeminipahungovercornchip
  • Reply 2 of 118
    Ouch, a lot of bad press for having changed the settings. I'm wondering if CR will say it was their mistake or Apple's?
    lostkiwiwatto_cobraredgeminipa
  • Reply 3 of 118
    This basically invalidates anything Consumer Reports does, test, or publishes in the future. Possibly in the past too. They knowingly changed settings that would not reflect an actual user's configuration, and deceitfully posted fraudulent results.

    If that's how they operate I have no interest in any other opinion they have to offer on any product.
    edited January 2017 spliff monkeypatchythepiratemobiusSpamSandwichpulseimagesrob53Nameo_tallest skilMacProlongpath
  • Reply 4 of 118
    I would guess that Consumer Reports did this knowingly. Their subscription base has likely declined as internet reviews of products are more immediate.
    Any chance to find something 'bad' about Apple really must be irresistible to them. I canceled my subscription many years ago, knowing that they are not the objective source they used to be.
    thewhitefalconspliff monkeydamn_its_hotpatchythepirateMikeymikemobiusSpamSandwichpulseimagesMacProDaekwan
  • Reply 5 of 118
    macxpressmacxpress Posts: 5,938member
    Why the hell would CR mess with any settings? Just take the damn thing out of the box and test it as is. I don't trust CR anyways. I'm sure they'll cover their ass and just blame it on Apple. I've always said these issues are most likely software issues with the battery. I seriously doubt Apple would release a laptop thats very well noted for its battery life with something else that gets half or worse the amount of battery life. Apple is one of the few you can actually rely on for accurate battery life. 
    spliff monkeypatchythepiratedamn_its_hotpalominelostkiwibadmonkwatto_cobraredgeminipacornchip
  • Reply 6 of 118
    Quote
    If Apple updates its software in a way that the company claims will substantively change battery performance, we will conduct fresh tests," the original report said.

    IF is the problem. CR could just as equally say, "Nope, we don't think Apple has done anything that will change our results. We stand by our results."

    The CR tests should be repeatable by anyone who chooses to perform them. If their original 'test steps' didn't state that they'd disabled the browser cache then CR deserve to get an awful lot of flack.
    Software and System testing is all about repeatability. AFAIK, the CR results were anything but that.

    lostkiwiwatto_cobraredgeminipacornchip
  • Reply 7 of 118
    Typical Consumer Reports.  They're not much different than other publications that benefit from click-bait headlines.  I would love to see the average age of their readers, because I'm guessing they're in the 55+ age bracket.  These are people who haven't figured out how to Google for product reviews.
    lostkiwiwatto_cobraredgeminipa
  • Reply 8 of 118
    Pretty clear CR was just looking for attention/headlines. No other reason to publish something with such widly inconsistent results. If they were reputable they would have contacted Apple before publishing to ensure there was nothing wrong with their testing before publishing.
    propodStrangeDayspatchythepiratedasanman69redgeminipaicoco3cornchip
  • Reply 9 of 118
    ceek74ceek74 Posts: 324member
    CR also found limited battery life when removing the battery.
    hydrogenboltsfan17jungmarkpatchythepirateSpamSandwichcyberzombiepulseimagesrob53ryan2012Daekwan
  • Reply 10 of 118
    Rayz2016Rayz2016 Posts: 6,957member
    Now will this discovery get the same traction in the IT press as the original duff report?

    I'm thinking not. 
    ai46pulseimagesbrucemclostkiwiwatto_cobraredgeminipa
  • Reply 11 of 118
    Here's the setting I think they are talking about. Would be good to know why CR turned it on. Probably to simulate real usage of the user visiting different websites. Need more info on this icon bug they are talking about to understand why it affected it, doesn't even say what icons they are talking about. I suppose if the icon bug caused 100% cpu use then yeh that would be a serious problem.

    welshdogfarjamedwatto_cobra
  • Reply 12 of 118
    Rayz2016Rayz2016 Posts: 6,957member

    kenc said:
    So CR turned off the browser cache...d'oh!
    Do they turn of the cache for all the machines they test? 


    bradmacpropscooter63baconstanganantksundaramlostkiwi
  • Reply 13 of 118
    jungmarkjungmark Posts: 6,927member
    Morons. So much for out of the box testing. 
    AI_liaslostkiwiwatto_cobraredgeminipa
  • Reply 14 of 118
    ben20ben20 Posts: 126member
    Whow ! Fake press. Recently with almost every purchasing decisicion, I felt the best and honest reviews come from people that actual use the product and are on a forum. Glad Apple resolved that, the Macbooks are awesome computers and to me the best money can buy.
    pulseimagesjuanguapowatto_cobraredgeminipa
  • Reply 15 of 118
    blastdoorblastdoor Posts: 3,570member
    I think CR's rationale for turing of cacheing is perfectly reasonable: 

    http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/

    I don't think anybody did anything wrong here -- neither Apple nor CR -- and both are handling it appropriately. 
    larryatzm41singularitydysamoriawelshdoganantksundaramsflagelfarjamedcrowleybeowulfschmidt
  • Reply 16 of 118
    There are a lot of complaints about this methodology in these comments, but my question is how would this differ from testing straight out of the box? On first test, the browser cache would not yet be populated and on subsequent tests it would, meaning they would see inconsistent results between 1st test and all later tests. They helped uncover a bug that would affect all Safari users until they had used the system enough that they regularly used the browser cache. Apple merely pointed out a work-around that would help them get consistent results.

    This shows the benefits of them working together - once the Safari bugfix is released to the general public, we can all expect better performance from these systems.
    tzm41dysamoriawelshdogsflagel
  • Reply 17 of 118
    Fixed???  I think not!  Let me share my experience that I went through:

    friday 11:25 (the 6th):
    On Friday I got to the Apple Store at 11:30 with no appointment. I was sent there by AppleCare due to the extreme nature of my problem with my new laptop just 15 days old.  I walked in with a fully charged battery. 100% AND stopped an employee so I'd have a witness. 

    I went because three nights in a row I took my laptop, bought my other half dinner at 6:45 then stayed to work till he got off at 10pm.  So I worked from 7pm till 10. Three nights in a row my computer went from 100% to 1% by 10pm (actually 2 hours and 56 minutes to be exact). 

    The odd thing is on Monday night I got in line at 2:00pm for a movie premier. By 4:57pm after leaving the house 100% it was 1% and I barely was able to save my work to my iPhone. Then it shut down. 

    My first reaction?  "I swear this was 100%. Maybe I'm on acid and trippin' ". LOL 

    In all 4 situactions I did not have WIFI or electricity so those options were not turned on. I also tried "graphics switching on" for 2 of the days, then  "graphics switching off"  the other two days. No improvement and it didn't get worse either. 

    So now it's Friday morning and at the apple store ... as I mentioned above. AppleCare sent me with no appointment. They made me wait for over two hours. That's ok. In this case that was a GOOD thing. I started logging in my battery drain using my iPhone. Check this out. Here's my battery while I waited:
    11:35.  100%
    11:48.  92%
    12:01.  83%
    12:08.  79%
    12;27.  66%
    12:46   52%
    12:57.  46%
    1:11.    38%
    1:18.    33%
    1:27.    27%
    1:34.    21%
    1:44.   15%

    WOW!  So here's the thing:  although we installed all our software on this Machine we only ran 2 apps and both were optimized for the touch bar model

    DJay 1.4.2
    iTunes 12.5.4

    That's it. This was to make sure we weren't running non optimized software. (I'm a beta tester for Apple so I went in to "beta tester" mode.). By limiting what I was using it helped control the experiment. One thing to note: the ONLY reason I installed my beta account was because with the golden master it shipped with I had serious graphics issues. Obviously this machine was shipped without a human launching it because it happened right after launching. Everyone EXCEPT Stevie Wonder and Helen Keller could have seen it. I barely got the beta installed because of the problems but once installed I've not had one graphic issue since. Thars good news I guess?

    So at the Apple Store while waiting the 2 hours I had iTunes open (files installed on computer no iCloud or internet) to see my playlists, but I was using DJay to test my crossfades on 350+ songs for an upcoming event. 

    I returned the unit through the online store the Genius Bar was not authorized to do anything except send it out for repairs r??  17 days old out for repair??  

    When i returned it not only did I leave EVERYTHING installed, and an authorization agreeing to them using it this way (I have nothing to hide)  I also sent the above letter and this statement:

    "I'm including this info because if I was receiving this laptop I would want to know these things that were done and the particulars to help diagnose the battery issue. 


    Screen dimmed to 12 clicks
    WIFI turned off and unavailable 
    Only two software titles running ever
    Pretty basic. 

    Hope this helps!  Call me if you need clarification"

    now apple claims safari caused it. HELLO?  We never launched safari or mail or anything else only those two programs. So you can stop spreading the "all clear" signal Apple!
    edited January 2017 SpamSandwichretrogustopulseimagesdysamoriaanantksundaramfarjamedhungover
  • Reply 18 of 118
    My take on Consumer Reports is they always seemed biased towards Window's machines over Apple.

    I just thought it was because they were engineers and a lot of engineers don't usually like Apple products for a variety of reasons. Mainly, b/c they can't "tinker" with Apple products the way they can with Windows boxes. Oh well. 
    patchythepirateretrogustowatto_cobra
  • Reply 19 of 118
    blastdoor said:
    I think CR's rationale for turing of cacheing is perfectly reasonable: 

    http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/

    I don't think anybody did anything wrong here -- neither Apple nor CR -- and both are handling it appropriately. 
    Thanks for posting, this completes the story.  I would expect the final republished results to equal or exceed Apple's stated battery life.
    farjamedwatto_cobra
  • Reply 20 of 118
    larryalarrya Posts: 608member
    Remember, guys, their testing consists of visiting a series of web sites over and over until the battery dies.  If that were my approach, I would definitely disable the cache. 
    tzm41dysamoriawelshdogclemynx
Sign In or Register to comment.