Apple says hidden Safari setting led to flawed Consumer Reports MacBook Pro battery tests

2456

Comments

  • Reply 21 of 118
    dewmedewme Posts: 5,335member
    This basically invalidates anything Consumer Reports does, test, or publishes in the future. Possibly in the past too. They knowingly changed settings that would not reflect an actual user's configuration, and deceitfully posted fraudulent results.

    If that's how they operate I have no interest in any other opinion they have to offer on any product.
    While I'm not a fan of Consumer Reports for several reasons, one flawed test case or specific test method used in one test scenario does not invalidate everything they've ever done or will ever do. That's just incredibly bogus logic. There is also no evidence whatsoever that CR was deceitful in any way. Crazy talk. What this does highlight however is the danger of reaching an opinion based on a single data point. It's always good to get a second opinion, and of course, to validate the test methodology if you're in the testing business. There's a good reason why critical systems are routinely subjected to independent verification and validation. The scientific method only works if the testing is valid. 
    edited January 2017 stompypscooter63dysamoriawelshdog
  • Reply 22 of 118
    volcanvolcan Posts: 1,799member
    blastdoor said:
    I think CR's rationale for turing of cacheing is perfectly reasonable: 
    Not really. Real world usage usually involves visiting one's favorite websites over and over during the day where caching is a big advantage. Some people probably visit their own Facebook page a hundred times a day.

    CR needs to develop better simulations.
    edited January 2017 watto_cobra
  • Reply 23 of 118
    What about the other browsers that CR tested?
  • Reply 24 of 118
    macxpressmacxpress Posts: 5,801member
    volcan said:
    blastdoor said:
    I think CR's rationale for turing of cacheing is perfectly reasonable: 
    Not really. Real world usage usually involves visiting one's favorite websites over and over during the day where caching is a big advantage. Some people probably visit Facebook a hundred times a day.

    CR needs to develop better simulations.
    CR just needs to be shut down, plain and simple! Its a shame people still rely on these fools for anything. Test things for yourself...don't buy what a magazine wants you to buy. Just because some pissant CR tester (expert) doesn't like something, doesn't mean you don't like it either. 
    watto_cobra
  • Reply 25 of 118
    This basically invalidates anything Consumer Reports does, test, or publishes in the future. Possibly in the past too. They knowingly changed settings that would not reflect an actual user's configuration, and deceitfully posted fraudulent results.

    If that's how they operate I have no interest in any other opinion they have to offer on any product.
    Yes, CR committed fraud here.

    It's not the first time.

    It won't be the last.

    I'm old enough to remember when CR was a reputable outfit.
    watto_cobra
  • Reply 26 of 118

    My take on Consumer Reports is they always seemed biased towards Window's machines over Apple.

    I just thought it was because they were engineers and a lot of engineers don't usually like Apple products for a variety of reasons. Mainly, b/c they can't "tinker" with Apple products the way they can with Windows boxes. Oh well. 
    I'm an engineer.  Engineers are like car guys-- there's a range, from the 'enthusiast" who is a kid who paints stripes on his car to make it a hot rod, thru the garage mechanic and to the guy who works for Ford GTO actually engineering fast cars.  IT's a huge range.


    Generally, the really skilled engineers want to use macs because they have something more important to do than screw around with their system.

    The wanna be and low skill engineers love linux because it gives them always a new problem to solve and feel smart.

    The non-engineers who think they are engineers become windows enthusiasts.
    StrangeDayspscooter63dysamoriaDaekwandewmebrucemcpulseimagesroundaboutnowwatto_cobra
  • Reply 27 of 118
    blastdoor said:
    I think CR's rationale for turing of cacheing is perfectly reasonable: 

    http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/

    I don't think anybody did anything wrong here -- neither Apple nor CR -- and both are handling it appropriately. 
    Um. Honestly. CR should have investigated their methodology before publishing if the results seemed skewed, unusual or off in any way. The fact that the problem was only evidenced in some of their test machines really should have been a red flag. As if Apple's stellar performance over many years of testing wasn't reason enough. To go ahead and publish the article before looking into errors in the test itself is a failure to do proper testing and frankly just BS to F'n the max. CR handled it very poorly. As a matter of fact the original article is still up on their website with no modification. To publish something after the fact on another page, not in print and at best will be tucked away in the magazine under some OPED after you just shat all over the product the month prior is pathetic. The cover for next month's CR should read "we don't know how to test products and even if the results are obviously out of whack with expectations we will do no further investigation into the errors in our testing before we publish because... well Click Bait. 

    Sorry CR did everything wrong here. 
    edited January 2017 Rayz2016StrangeDayspscooter63Daekwanbrucemcwatto_cobrahungovermobius
  • Reply 28 of 118
    Here's the setting I think they are talking about. Would be good to know why CR turned it on. Probably to simulate real usage of the user visiting different websites. Need more info on this icon bug they are talking about to understand why it affected it, doesn't even say what icons they are talking about. I suppose if the icon bug caused 100% cpu use then yeh that would be a serious problem.

    This setting is used when testing page reload times to reflect modem, cpu, and download speeds without using the cache, which would be much faster and less resource consuming.
    hungover
  • Reply 29 of 118
    larryjwlarryjw Posts: 1,031member
    Browsers other than Safari didn't show battery life problems and inconsistencies.

    So, we need a complete accounting here. Measure battery life in a two dimensional grid of browser x cache/no cache.


  • Reply 30 of 118
    netroxnetrox Posts: 1,415member
    It is really odd to turn off the web cache. I have the developer tool enabled but the cache is still on and I see no reason to turn it off except for testing - it takes a lot of bandwidth to retrieve files and adds more time to browsing. Why would it Developer Tool be "default" for CR testing since most consumers don't even know about it? But I am surprised that no cache would make a dramatic difference in battery life. Looking forward to the update for Safari bug.
  • Reply 31 of 118
    tzm41tzm41 Posts: 95member
    It is wonderful how so many people here are bashing CR. CR used the same testing methodology and ranked the past MB models high. CR also used the same methodology for all other laptops without this kind of jarring finding.
    Reasonably, users other than web developers would not turn this option on. But they would not reload the same page ten thousand times before the battery dies either. This setting is just to mimic a use case when the user is browsing different websites, and is completely fair since CR uses it across all laptops.
    Apple admitted it is a bug in Safari that cause the fluctuation, but people are blaming CR for using the browser wrong. To me, apple fanboys are saying when CR recommends MBs it is trustworthy, when it does not it is crap.
    stompylorin schultzteaearlegreyhotsingularitydysamoriawilliamlondonhungover
  • Reply 32 of 118
    wigginwiggin Posts: 2,265member
    Quote
    If Apple updates its software in a way that the company claims will substantively change battery performance, we will conduct fresh tests," the original report said.

    IF is the problem. CR could just as equally say, "Nope, we don't think Apple has done anything that will change our results. We stand by our results."

    The CR tests should be repeatable by anyone who chooses to perform them. If their original 'test steps' didn't state that they'd disabled the browser cache then CR deserve to get an awful lot of flack.
    Software and System testing is all about repeatability. AFAIK, the CR results were anything but that.

    What would really be interesting is to repeat the test both with and without the cache disabled after Apple's bug fix is released. Remember, the article also stated that this also led Apple to discover a bug in their own code that made the results even worse. If CR's test script is set up to repeatedly load the same page as a simulation of random web browsing, then having the cache enabled would essentially be cheating in favor of better battery performance as the entire page would already be in the cache. Perhaps this test protocol, including disabling the cache, worked fine in past iterations of testing with other Macs, but when combined with this Apple bug it was no longer a valid test.

    Clearly CR needs to update their testing protocol, but I'm not sure I'm ready to go so far as to say it was malicious as some here are suggesting. Keep in mind, there have been many other reports of questionable battery life reported by other sources. Perhaps this bug Apple discovered in their own code contributed to those other reported experiences as well. Apple has previously stated that there was no software issues related to battery life. This incident has proven that to be false.

    And I hope this bug they are reportedly fixing also fixes the issue myself and others have reported with Sierra on earlier models (mine is a 2015 MBP). Something in Sierra is causing the discreet GPU to engage when it's not needed and was never engaged in previous OS versions. For example, opening the color picker in TextEdit activates the discreet GPU in Sierra, and it stays active until you exit TextEdit. Launching Quicken 2007 also engages the GPU in Sierra but not in any previous OS version. Quicken I could maybe accept as it's obviously older software that might not have the correct configuration. It was coded before there was automatic graphics switching in Mac laptops, but it did behave properly in OSes prior to Sierra. And that doesn't explain TextEdit which was released with Sierra.
    lorin schultzwelshdog
  • Reply 33 of 118
    The issue that made CR successful no longer exists.

    From 1932 until 1940 consumer demeanor demand could not be satisfied.  The Depression destroyed consumer buying power resulting in factory closures.  From 1941 through 1945 (WWII) consumer demand exploded but nothing consumer was being produced (war effort).  It took about 2 years to convert our wartime economy to a peace time economy.  From then on US manufacturers could not satisfy all that pent up consumer demand.  The larger retailers employed in-house repair facilities TO REPAIR BRAND NEW PRODUCTS BEFORE PUTTIMG THEM ON THE FLOOR.

    US manufacturing quality was dismal.  This was overlooked by consumers having no other options (European and Japanese manufacturing were still recovering from WWII).

    This condition continued until the early 1960s in the consumer electronics market when Japan began exporting radios, TVs, etc. ,that "just worked".  It took another 20 years before Detroit, faced with extinction, began improving the quality of its products.

    From the end of WWII until the early 1990s the market was ripe for an unbiased authority that test and recommend products.  CR filled and owned that niche.

    For the last 25 years or so, CR has become increasingly irreverent as US manufacturers have been replaced by offshore manufacturing or have adopted a "Japanese" quality ethos.  The market niche that created CR has passed, it will only be a matter of time before CR does too.
    Rayz2016StrangeDayspscooter63pulseimages
  • Reply 34 of 118
    sdw2001sdw2001 Posts: 18,015member
    Just shows that CR's tests are worthless in terms of applicability to the real world.  This is a setting literally zero consumers would change.  It's not even visible.   

    Told. You. So. 

    hungover
  • Reply 35 of 118
    wigginwiggin Posts: 2,265member
    Typical Consumer Reports.  They're not much different than other publications that benefit from click-bait headlines.  I would love to see the average age of their readers, because I'm guessing they're in the 55+ age bracket.  These are people who haven't figured out how to Google for product reviews.
    "click-bait"? Um, you do realize that CR doesn't sell advertising either in their magazine or their website, right? Perhaps it might get a small number of people to subscribe if they visit their web site, but I'd hardly call it click-bait.
    dysamorianocreatorblue
  • Reply 36 of 118
    larrya said:
    Remember, guys, their testing consists of visiting a series of web sites over and over until the battery dies.  If that were my approach, I would definitely disable the cache. 
    Exactly. Any kind of battery benchmark will be inaccurate by design — people do not load one web page every 30 seconds and then go on to another one, for example. A benchmark that loaded pages by random intervals would need to be run hundreds of times in order to get a meaningful distribution of data, we'd still be waiting for CR to publish a damn review.

    Benchmarking with caches enabled would have unpredictable results based on when the websites in question updated (say the NYTimes front page is in the roster, and a new issue with cache-busting changes pops up in the middle of one benchmark but not another) and also how caching is configured for the various assets… also shared assets like the cached jQuery in GoogleAPIs. The best way to get "clean" data is to just disable caching and accept that as a factor of your test. The test will probably show slightly worse battery life than real world use, but the results of the test will be consistent and only need to be run dozens of times instead of hundreds.

    Apple and Consumer Reports both did their jobs correctly

    ------

    Personally, I have found the battery life of this 15" Touchbar MBP to be just slightly better than the 13" Retina MBP it's replacing, but one thing the reviews left out is that the charging is unbelievably fast. If I leave this plugged in for 20-30 minutes for lunch or to take a walk, I can easily get through an entire work day. My jury is still out on the Touchbar but I sure like the rest of this thing.
    edited January 2017 singularitypscooter63dysamoriapulseimageswilliamlondon
  • Reply 37 of 118
    I used to subscribe to Consumer Reports... before quick user-ratings and in-depth reviews of most any major product could be found on the Internet... but when I stopped subscribing, one of the reasons was CR's strange antipathy for Apple. Obvious to me, some CR head of technology or electronics testing was a PC guy with an attitude, which Consumer Reports shouldn't have. Their low or muted Apple scores were always a head-scratcher to me. And now it seems like there was internal CR sabotage for the MacBook Pro review.
    edited January 2017
  • Reply 38 of 118
    Rayz2016Rayz2016 Posts: 6,957member
    blastdoor said:
    I think CR's rationale for turing of cacheing is perfectly reasonable: 

    http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/

    I don't think anybody did anything wrong here -- neither Apple nor CR -- and both are handling it appropriately. 

    When this report first came out, I was happy to give them the benefit of the doubt, even though it was plainly obvious that the problem was with Safari, not the Macbook. Why? Because a consumer wouldn't know to separate one from the other, and so from a consumer's point of view it was a valid test.

    Unfortunately, once they started fiddling with debug settings on Safari, it stopped being a consumer laptop test and became a test to see how Safari reacted to being bombarded with identical web pages over several hours. This was supposed to be a test of what the user would experience out of the box. It wasn't. And to be honest, they shouldn't be disabling the caches on any machines because that is not what consumers do. They really need to rethink their testing.


    "We can't recommend this washing machine because we found that the spin cycle was not as fast as claimed."

    What they failed to mention is that their tests only work if the washing machine is connected to a car battery.

    Still, the good news is that Apple has a fix, and the folk hoping this was hardware problem get to be disappointed.

    pscooter63
  • Reply 39 of 118
    StrangeDaysStrangeDays Posts: 12,844member
    razormaid said:
    Fixed???  I think not!  Let me share my experience that I went through:

    friday 11:25 (the 6th):
    On Friday I got to the Apple Store at 11:30 with no appointment. I was sent there by AppleCare due to the extreme nature of my problem with my new laptop just 15 days old.  I walked in with a fully charged battery. 100% AND stopped an employee so I'd have a witness. 

    I went because three nights in a row I took my laptop, bought my other half dinner at 6:45 then stayed to work till he got off at 10pm.  So I worked from 7pm till 10. Three nights in a row my computer went from 100% to 1% by 10pm (actually 2 hours and 56 minutes to be exact). 

    The odd thing is on Monday night I got in line at 2:00pm for a movie premier. By 4:57pm after leaving the house 100% it was 1% and I barely was able to save my work to my iPhone. Then it shut down. 

    My first reaction?  "I swear this was 100%. Maybe I'm on acid and trippin' ". LOL 

    In all 4 situactions I did not have WIFI or electricity so those options were not turned on. I also tried "graphics switching on" for 2 of the days, then  "graphics switching off"  the other two days. No improvement and it didn't get worse either. 

    So now it's Friday morning and at the apple store ... as I mentioned above. AppleCare sent me with no appointment. They made me wait for over two hours. That's ok. In this case that was a GOOD thing. I started logging in my battery drain using my iPhone. Check this out. Here's my battery while I waited:
    11:35.  100%
    11:48.  92%
    12:01.  83%
    12:08.  79%
    12;27.  66%
    12:46   52%
    12:57.  46%
    1:11.    38%
    1:18.    33%
    1:27.    27%
    1:34.    21%
    1:44.   15%

    WOW!  So here's the thing:  although we installed all our software on this Machine we only ran 2 apps and both were optimized for the touch bar model

    DJay 1.4.2
    iTunes 12.5.4

    That's it. This was to make sure we weren't running non optimized software. (I'm a beta tester for Apple so I went in to "beta tester" mode.). By limiting what I was using it helped control the experiment. One thing to note: the ONLY reason I installed my beta account was because with the golden master it shipped with I had serious graphics issues. Obviously this machine was shipped without a human launching it because it happened right after launching. Everyone EXCEPT Stevie Wonder and Helen Keller could have seen it. I barely got the beta installed because of the problems but once installed I've not had one graphic issue since. Thars good news I guess?

    So at the Apple Store while waiting the 2 hours I had iTunes open (files installed on computer no iCloud or internet) to see my playlists, but I was using DJay to test my crossfades on 350+ songs for an upcoming event. 

    I returned the unit through the online store the Genius Bar was not authorized to do anything except send it out for repairs r??  17 days old out for repair??  

    When i returned it not only did I leave EVERYTHING installed, and an authorization agreeing to them using it this way (I have nothing to hide)  I also sent the above letter and this statement:

    "I'm including this info because if I was receiving this laptop I would want to know these things that were done and the particulars to help diagnose the battery issue. 

    Screen dimmed to 12 clicks
    WIFI turned off and unavailable 
    Only two software titles running ever
    Pretty basic. 

    Hope this helps!  Call me if you need clarification"

    now apple claims safari caused it. HELLO?  We never launched safari or mail or anything else only those two programs. So you can stop spreading the "all clear" signal Apple!
    So you're suggesting that whatever issue you have with your copy (which may be defective) is related to what Apple and CR say was the issue in their testing? I'm not seeing the connection.
    pscooter63pulseimagesroundaboutnow
  • Reply 40 of 118
    tzm41 said:
    This setting is just to mimic a use case when the user is browsing different websites, and is completely fair since CR uses it across all laptops.
    Exactly what I was thinking. Thanks!
    AI_lias
Sign In or Register to comment.