Apple says hidden Safari setting led to flawed Consumer Reports MacBook Pro battery tests
Battery tests conducted by Consumer Reports showing inconsistent uptime with Apple's new MacBook Pro were inaccurate, Apple said on Tuesday, explaining that the publication had enabled a hidden setting in its Safari browser.
In a statement issued to AppleInsider, Apple said it worked with Consumer Reports over the holidays, and determined that its testing methods for the new MacBook Pro were flawed. The company had initially announced just before Christmas that it was cooperating with Consumer Reports to determine what went wrong with the MacBook Pro units they tested.
"We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache," Apple's statement reads. "This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab.
"After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life. We have also fixed the bug uncovered in this test. This is the best pro notebook we've ever made, we respect Consumer Reports and we're glad they decided to revisit their findings on the MacBook Pro."
The Safari fix is available to beta testers, presumably in the latest pre-release build of macOS 10.12.3, made available to registered developers on Tuesday. Apple says the full fix will be provided to the public in an upcoming macOS update.
Previous testing by the publication led Consumer Reports to not recommend any new MacBook Pro model due to battery life concerns. Their in-house testing revealed wild fluctuations in unplugged operating survivability, in some cases ranging from 16 hours to as little as 3.75 hours.
In its initial findings, the publication left the door open for an Apple response, noting the battery life of many modern products are "influenced" by software updates.
"If Apple updates its software in a way that the company claims will substantively change battery performance, we will conduct fresh tests," the original report said.
In a statement issued to AppleInsider, Apple said it worked with Consumer Reports over the holidays, and determined that its testing methods for the new MacBook Pro were flawed. The company had initially announced just before Christmas that it was cooperating with Consumer Reports to determine what went wrong with the MacBook Pro units they tested.
Apple says battery inconsistency was caused by a hidden Safari setting for developers which turns off the browser cache. The setting is not used by customers.
"We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache," Apple's statement reads. "This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab.
"After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life. We have also fixed the bug uncovered in this test. This is the best pro notebook we've ever made, we respect Consumer Reports and we're glad they decided to revisit their findings on the MacBook Pro."
The Safari fix is available to beta testers, presumably in the latest pre-release build of macOS 10.12.3, made available to registered developers on Tuesday. Apple says the full fix will be provided to the public in an upcoming macOS update.
Previous testing by the publication led Consumer Reports to not recommend any new MacBook Pro model due to battery life concerns. Their in-house testing revealed wild fluctuations in unplugged operating survivability, in some cases ranging from 16 hours to as little as 3.75 hours.
In its initial findings, the publication left the door open for an Apple response, noting the battery life of many modern products are "influenced" by software updates.
"If Apple updates its software in a way that the company claims will substantively change battery performance, we will conduct fresh tests," the original report said.
Comments
If that's how they operate I have no interest in any other opinion they have to offer on any product.
Any chance to find something 'bad' about Apple really must be irresistible to them. I canceled my subscription many years ago, knowing that they are not the objective source they used to be.
If Apple updates its software in a way that the company claims will substantively change battery performance, we will conduct fresh tests," the original report said.
IF is the problem. CR could just as equally say, "Nope, we don't think Apple has done anything that will change our results. We stand by our results."
The CR tests should be repeatable by anyone who chooses to perform them. If their original 'test steps' didn't state that they'd disabled the browser cache then CR deserve to get an awful lot of flack.
Software and System testing is all about repeatability. AFAIK, the CR results were anything but that.
I'm thinking not.
Do they turn of the cache for all the machines they test?
http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/
I don't think anybody did anything wrong here -- neither Apple nor CR -- and both are handling it appropriately.
This shows the benefits of them working together - once the Safari bugfix is released to the general public, we can all expect better performance from these systems.
friday 11:25 (the 6th):
On Friday I got to the Apple Store at 11:30 with no appointment. I was sent there by AppleCare due to the extreme nature of my problem with my new laptop just 15 days old. I walked in with a fully charged battery. 100% AND stopped an employee so I'd have a witness.
I went because three nights in a row I took my laptop, bought my other half dinner at 6:45 then stayed to work till he got off at 10pm. So I worked from 7pm till 10. Three nights in a row my computer went from 100% to 1% by 10pm (actually 2 hours and 56 minutes to be exact).
The odd thing is on Monday night I got in line at 2:00pm for a movie premier. By 4:57pm after leaving the house 100% it was 1% and I barely was able to save my work to my iPhone. Then it shut down.
My first reaction? "I swear this was 100%. Maybe I'm on acid and trippin' ". LOL
In all 4 situactions I did not have WIFI or electricity so those options were not turned on. I also tried "graphics switching on" for 2 of the days, then "graphics switching off" the other two days. No improvement and it didn't get worse either.
So now it's Friday morning and at the apple store ... as I mentioned above. AppleCare sent me with no appointment. They made me wait for over two hours. That's ok. In this case that was a GOOD thing. I started logging in my battery drain using my iPhone. Check this out. Here's my battery while I waited:
11:35. 100%
11:48. 92%
12:01. 83%
12:08. 79%
12;27. 66%
12:46 52%
12:57. 46%
1:11. 38%
1:18. 33%
1:27. 27%
1:34. 21%
1:44. 15%
WOW! So here's the thing: although we installed all our software on this Machine we only ran 2 apps and both were optimized for the touch bar model
DJay 1.4.2
iTunes 12.5.4
That's it. This was to make sure we weren't running non optimized software. (I'm a beta tester for Apple so I went in to "beta tester" mode.). By limiting what I was using it helped control the experiment. One thing to note: the ONLY reason I installed my beta account was because with the golden master it shipped with I had serious graphics issues. Obviously this machine was shipped without a human launching it because it happened right after launching. Everyone EXCEPT Stevie Wonder and Helen Keller could have seen it. I barely got the beta installed because of the problems but once installed I've not had one graphic issue since. Thars good news I guess?
So at the Apple Store while waiting the 2 hours I had iTunes open (files installed on computer no iCloud or internet) to see my playlists, but I was using DJay to test my crossfades on 350+ songs for an upcoming event.
I returned the unit through the online store the Genius Bar was not authorized to do anything except send it out for repairs r?? 17 days old out for repair??
When i returned it not only did I leave EVERYTHING installed, and an authorization agreeing to them using it this way (I have nothing to hide) I also sent the above letter and this statement:
"I'm including this info because if I was receiving this laptop I would want to know these things that were done and the particulars to help diagnose the battery issue.
Screen dimmed to 12 clicks
WIFI turned off and unavailable
Only two software titles running ever
Pretty basic.
Hope this helps! Call me if you need clarification"
now apple claims safari caused it. HELLO? We never launched safari or mail or anything else only those two programs. So you can stop spreading the "all clear" signal Apple!
I just thought it was because they were engineers and a lot of engineers don't usually like Apple products for a variety of reasons. Mainly, b/c they can't "tinker" with Apple products the way they can with Windows boxes. Oh well.