Apple says hidden Safari setting led to flawed Consumer Reports MacBook Pro battery tests

1356

Comments

  • Reply 41 of 118
    This basically invalidates anything Consumer Reports does, test, or publishes in the future. Possibly in the past too.
    They stopped being a legitimate outlet after denouncing the iPhone 4.
    nhtpscooter63
  • Reply 41 of 118
    To those bashing Consumer Reports for what they perceive to be flawed testing:

    1. OF COURSE they disabled caching. It's a test of battery use while surfing. To make the test consistent and repeatable across multiple runs and various models, the test involves having the machines continuously load content from a fixed set of sources. There wouldn't be any point to the test if it only involved downloading the content once then storing it.

    2. The results VARIED. This wasn't a case of an artificial test that doesn't reflect real-world usage making the battery life seem worse than it would be in actual use, it was a case of sometimes the battery lasting a long time and other times running dead very quickly. Since the test was the same every time, the huge variations in results were a legitimate cause for concern, REGARDLESS of the testing method.

    3. According to APPLE, not CR, the problem is the result of a BUG, not the testing methodology per se. The testing method merely exposed it. How long would it have gone on unnoticed and unfixed if the test hadn't uncovered it? It may not have affected me or you, but obviously some people would suffer from it. This is a positive outcome.

    4. Reports from others, including participants in this forum, outlining how their real-world use of the machines has resulted in excessive battery drain, show that the problem exists outside the circumstances imposed by CR's testing. CR did NOT run a flawed test and report erroneous findings. They reported, as have others, that there are still wrinkles to be ironed out.

    5. CR didn't say it's a crappy machine. They said it's a great machine with a really big "something's not right here." They were absolutely CORRECT.

    I'm lucky that I haven't had battery problems (so far, knock wood), but others obviously have, and depending on what's causing them, maybe it's only because I haven't yet done the things that cause the problem to surface and later WILL have a problem. I welcome any information that helps lead to a cure.
    AQdysamoriablastdoornetroxtzm41pulseimagesroundaboutnowstompyhucom2000nocreatorblue
  • Reply 43 of 118
    MacProMacPro Posts: 19,718member
    Defamation?
  • Reply 44 of 118
    This basically invalidates anything Consumer Reports does, test, or publishes in the future. Possibly in the past too. They knowingly changed settings that would not reflect an actual user's configuration, and deceitfully posted fraudulent results.

    If that's how they operate I have no interest in any other opinion they have to offer on any product.
    I doubt they did it maliciously. But I don't doubt that they should have never released results that were so wildly inconsistent with the experiences of tens of thousands of users without first checking with Apple to find out what was obviously wrong with their test. Better late than never I guess. Hope they learn from this. 
    pscooter63
  • Reply 45 of 118
    nhtnht Posts: 4,522member
    tzm41 said:
    It is wonderful how so many people here are bashing CR. CR used the same testing methodology and ranked the past MB models high. CR also used the same methodology for all other laptops without this kind of jarring finding.
    Reasonably, users other than web developers would not turn this option on. But they would not reload the same page ten thousand times before the battery dies either. This setting is just to mimic a use case when the user is browsing different websites, and is completely fair since CR uses it across all laptops.
    Apple admitted it is a bug in Safari that cause the fluctuation, but people are blaming CR for using the browser wrong. To me, apple fanboys are saying when CR recommends MBs it is trustworthy, when it does not it is crap.
    The reason that folks are giving CR a hard time is that they didn't see the issue in Chrome but then labeled it as a problem of the laptop itself rather than potentially a problem with Safari.

    "Once our official testing was done, we experimented by conducting the same battery tests using a Chrome browser, rather than Safari. For this exercise, we ran two trials on each of the laptops, and found battery life to be consistently high on all six runs."

    The CR battery test is also stupid as it only measures battery life of web surfing and not say watching a movie (video loop) or playing games (game benchmark loop) or content creation (hand break or photoshop benchmark loop).  Had their battery test not been so stupid and covered more than one use case it would have been equally obvious that something was wrong with the web benchmark and not the laptop itself.

    That Chrome posted consistently high battery tests and they went with a sensationalist headline and result indicates that CR's intent was to generate controversy and increase their exposure.
    entropyspscooter63williamlondon
  • Reply 46 of 118
    Typical Consumer Reports.  They're not much different than other publications that benefit from click-bait headlines.  I would love to see the average age of their readers, because I'm guessing they're in the 55+ age bracket.  These are people who haven't figured out how to Google for product reviews.
    Yeah, I'm 55+, old enough to have been an Apple fan and user since the first MacIntosh. And to have bought AAPL when everyone else said I was crazy, and profited handsomely from it. Guess that makes me old and stupid. 
    dysamoriapscooter63nocreatorblue
  • Reply 47 of 118
    AQAQ Posts: 1member
    Makes perfect sense to disable the cache. CR is testing a browser retrieving content from a server, not local files. The animosity from my fellow Apple fanboys is disappointing.
    dysamoriablastdoortzm41singularityhucom2000nocreatorbluewilliamlondonbeowulfschmidt
  • Reply 48 of 118
    irelandireland Posts: 17,798member
    This basically invalidates anything Consumer Reports does, test, or publishes in the future.
    I disagree. Everyone makes mistakes. Take it on the chin and move on. They agreed to help figure it out with Apple and they retested and they uncovered a genuine bug. Disagree with your assessment.
    edited January 2017 dysamoria
  • Reply 49 of 118
    irelandireland Posts: 17,798member

    Typical Consumer Reports.  They're not much different than other publications that benefit from click-bait headlines.
    I didn't know this. I nearly care too ha.
  • Reply 50 of 118
    SoliSoli Posts: 10,035member
    As expected, it was a bug in the OS (yes, Safari is an app but it's part of the OS), and CR caused the issue with how they conducted their tests. CR should've known better and contacted Apple when they saw such wild results between repeated testing.
    brucemcpscooter63pulseimagesroundaboutnow
  • Reply 51 of 118
    kenc said:
    So CR turned off the browser cache...d'oh!
    Doh? They do it to simulate browsing many websites (not the same ones over and over again) - this is standard for testing battery life or even browser performance and comparing with other machines with the same settings. There was a bug in Safari that meant some images were rendered over over again killing battery life.

    If you read the article (challenging I know) you'll notice Apple has issued a fix for this bug. CR will not change their testing approach.
    singularitynocreatorblue
  • Reply 52 of 118
    nhtnht Posts: 4,522member
    This basically invalidates anything Consumer Reports does, test, or publishes in the future. Possibly in the past too.
    They stopped being a legitimate outlet after denouncing the iPhone 4.
    Yes.
    pscooter63williamlondon
  • Reply 53 of 118
    blastdoorblastdoor Posts: 3,258member
    AQ said:
    Makes perfect sense to disable the cache. CR is testing a browser retrieving content from a server, not local files. The animosity from my fellow Apple fanboys is disappointing.
    I totally agree. 

    Everyone has handled this well except for the rabid fanboys. 
    singularitynocreatorbluewilliamlondon
  • Reply 54 of 118
    netroxnetrox Posts: 1,415member
    kenc said:
    So CR turned off the browser cache...d'oh!
    Doh? They do it to simulate browsing many websites (not the same ones over and over again) - this is standard for testing battery life or even browser performance and comparing with other machines with the same settings. There was a bug in Safari that meant some images were rendered over over again killing battery life.

    If you read the article (challenging I know) you'll notice Apple has issued a fix for this bug. CR will not change their testing approach.
    No, by default, all browsers cache resources. I don't know of any browsers that do not cache resources by default. 
      But I am just surprised how disabled cache would dramatically shorten a battery life.  Exactly what is the mechanism that a disabled cache feature would have on battery?

    pulseimages
  • Reply 55 of 118
    Typical Consumer Reports.  They're not much different than other publications that benefit from click-bait headlines.  I would love to see the average age of their readers, because I'm guessing they're in the 55+ age bracket.  These are people who haven't figured out how to Google for product reviews.
    Yeah, I'm 55+, old enough to have been an Apple fan and user since the first MacIntosh. And to have bought AAPL when everyone else said I was crazy, and profited handsomely from it. Guess that makes me old and stupid. 
    I'm not yet 55, which must be why I've figured out how to Google for product reviews. What I've learned from those:

    1. Sometimes they're really handy.

    2. Most of the time they're written by someone with an axe to grind who denounces the item in question on the basis of a single issue.

    3. Understanding of "scientific method" is on life support on this planet. Controlling variables, consistent methodology, valid measurement methods, and repeatable results are concepts that either elude most people or are just more effort than they're willing to invest. The result is wildly conflicting reviews with some saying the product is proof of the existence of God and others saying it will cause suicide.

    4. Institutionalized testing usually -- not always -- does a better of job of providing a valid basis for comparison. Like fuel efficiency scores. Even though everyone understands that the "miles per gallon" reported isn't an accurate reflection of what a vehicle will actually get in real life, at least we know that everyone is using the same testing method so we can make useful comparisons. Whether or not a vehicle actually GETS 50 mpg like it says on the sticker doesn't matter -- we know that the same testing was used for the one that reports 10 mpg so we can conclude that the first one is probably more fuel efficient. You don't get that consistency with random reviews.
    tokyojimupscooter63roundaboutnowbeowulfschmidtrobin huber
  • Reply 56 of 118
    croprcropr Posts: 1,122member
    If I would be doing these tests, I would also switch off the cache.  It seems to me quite logical that I want to have real traffic on the wifi in order to mimic a realistic user behaviour.
    Anyhow it is strange to me that there is such a huge impact on battery life.

    beowulfschmidt
  • Reply 57 of 118
    What happen with the other people with the same battery problem? 
    pulseimages
  • Reply 58 of 118
    SoliSoli Posts: 10,035member
    netrox said:
    Exactly what is the mechanism that a disabled cache feature would have on battery?
    As stated in the article, instead of simply loading an image once for a page, it would continually reload them. Not sure if this was all types or sizes, or just certain ones, but querying a server to continually get the info over and over will drain the battery.

    If CR had monitored the amount of data being transmitted and received with their tests they would've seen that something was off, but they should've known something was off by the wild results they experienced.

    Additionally, because Apple now knows they disable caching, which is an atypical option for users, they can not only resolve this bug but build in methods that will further optimize for battery life over performance for their future tests so I wouldn't be shocked to see CR's future Mac testing exceeding Apple's results even more than they have in the past.
  • Reply 59 of 118
    StrangeDaysStrangeDays Posts: 12,834member

    3. According to APPLE, not CR, the problem is the result of a BUG, not the testing methodology per se. The testing method merely exposed it. How long would it have gone on unnoticed and unfixed if the test hadn't uncovered it? It may not have affected me or you, but obviously some people would suffer from it. This is a positive outcome.

    Actually I think Apple blamed both the methodology and the bug. See their statement:

    “We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug..."

    Note that they use the word "also triggered" the obscure bug. "Also", not "subsequently" triggered the bug. Right or wrong, they seem to be saying both conditions are responsible for the results.
    edited January 2017 Solipscooter63pulseimages
  • Reply 60 of 118
    MplsPMplsP Posts: 3,911member
    I agree that turning off the cache for testing purposes was a reasonable thing to do if they were trying to simulate surfing different web sites. If they did this on every computer they tested then it's hard to argue that they're singling out Apple. The fact that it was so inconsistent and only occurred on Safari and not Chrome should have been a red flag to them that it was a software bug, not a hardware issue. Even if this was not a 'real world' test in that respect, the issue was ultimately caused by a software bug in Safari. I would hope that CR puts a link to their repeat tests on the page with the original article and issues a correction as that would be the right thing to do from a journalistic standpoint.

    As far as the other people experiencing battery issues, it's impossible to say what they're from. It's doubtful that they're all caused by this bug (I suppose we can hope, right?) but this is an excellent example of how bad software design or software bugs can dramatically alter hardware performance. My hope is that the issues people are experiencing are actually software related and can be remedied with a simple software update. They would still be a black eye for Apple, but at least a much smaller one that can be fixed.
Sign In or Register to comment.