Pretty reasonable thing for CR to do and unfortunate for both CR and Apple that they ran into a bug which caused such a kerfuffle.
Anyone flinging vitriol to blame either side needs to go spend some time outside for a change.
Why shouldn't there be blame. Is having a bug in a SW not the result of Apple's software developers making a mistake? And doesn't a wide a range of atypical results ranging from about 4.5 to 19.5 hours indicate that CR should have questioned something about their test? Not necessarily a problem with their methodology, which is evidence by previous Mac tests and with tests with Chrome, but with something unique that should have brought them to a conclusion that there's a bug in Safari that's causing an issue for their tests.
Haven't they contacted manufacturers in the past when something they were testing wasn't working properly? And this isn't even a HW issue (although that was possible).
Pretty clear CR was just looking for attention/headlines. No other reason to publish something with such widly inconsistent results. If they were reputable they would have contacted Apple before publishing to ensure there was nothing wrong with their testing before publishing.
I agree. If you look at the original article online, you will see a plenty of anti-Apple sentiment in the comments section. Over the past few years I've tired reading their glowing reviews of Samsung/Android phones and pushing Apple's down in the ratings because they don't have expandable memory, wireless charging, etc. No mention of dearth of Android updates, and not much about malware on Android. I get the feeling that the testing staff is in bed with Android. For an objective testing publication, this is not good. I just cancelled my subscription.
Consumer Reports explains their testing methodology as essentially an attempt to reduce variables across different test subjects. I get that, but disagree that it offers an equitable comparison. (As an aside, by switching off the browser cache in an effort to reduce testing variables, CR apparently created a variable missed during Apple's own testing of the product, triggering the issue with reloading icons. Nice catch, but they found an obscure bug, not a fundamental flaw...) In their methodology explanation, CR notes that various manufacturers will use different techniques to enhance battery life, such as dimming the screen, the browser cache, etc. So to 'even things out,' they disable all those things for their tests.
For Apple, since they design the OS in conjunction with the hardware, it's extremely likely they're taking all sorts of opportunities to tweak things to improve performance. That's why comparing raw hardware stats for memory, CPU, bus, etc., doesn't offer a proper comparison to other computers. So in effect, CR is de-engineering the devices before they test them. They think they're limiting variables, but they're not. They're adding back in variables that the computers' designers had engineered out. Under this methodology, a manufacturer that ham-fists the rest of the design and software, but shoves an enormous battery into a thicker notebook, would perform better than a MBP with a smaller battery operating a much more efficient machine. That's not a good test.
Imagine producing a powerful, fuel-efficient car, and getting a 'not recommended' review from CR. You ask what happened, only to learn that CR removed the turbo (something standard in the design of your car), so that it could be compared to other cars that don't have a turbo, and they subsequently found your car to be underpowered and much less fuel efficient than advertised. Is that a good or fair test? No, it's not.
Spot on. CR's test approach was well intentioned and free of malice, but it was crude, naive, and lacking in sound engineering and testing principles. They should learn from this experience.
Fixed??? I think not! Let me share my experience that I went through:
friday 11:25 (the 6th): On Friday I got to the Apple Store at 11:30 with no appointment. I was sent there by AppleCare due to the extreme nature of my problem with my new laptop just 15 days old. I walked in with a fully charged battery. 100% AND stopped an employee so I'd have a witness.
I went because three nights in a row I took my laptop, bought my other half dinner at 6:45 then stayed to work till he got off at 10pm. So I worked from 7pm till 10. Three nights in a row my computer went from 100% to 1% by 10pm (actually 2 hours and 56 minutes to be exact).
The odd thing is on Monday night I got in line at 2:00pm for a movie premier. By 4:57pm after leaving the house 100% it was 1% and I barely was able to save my work to my iPhone. Then it shut down.
My first reaction? "I swear this was 100%. Maybe I'm on acid and trippin' ". LOL
In all 4 situactions I did not have WIFI or electricity so those options were not turned on. I also tried "graphics switching on" for 2 of the days, then "graphics switching off" the other two days. No improvement and it didn't get worse either.
So now it's Friday morning and at the apple store ... as I mentioned above. AppleCare sent me with no appointment. They made me wait for over two hours. That's ok. In this case that was a GOOD thing. I started logging in my battery drain using my iPhone. Check this out. Here's my battery while I waited: 11:35. 100% 11:48. 92% 12:01. 83% 12:08. 79% 12;27. 66% 12:46 52% 12:57. 46% 1:11. 38% 1:18. 33% 1:27. 27% 1:34. 21% 1:44. 15%
WOW! So here's the thing: although we installed all our software on this Machine we only ran 2 apps and both were optimized for the touch bar model
DJay 1.4.2 iTunes 12.5.4
That's it. This was to make sure we weren't running non optimized software. (I'm a beta tester for Apple so I went in to "beta tester" mode.). By limiting what I was using it helped control the experiment. One thing to note: the ONLY reason I installed my beta account was because with the golden master it shipped with I had serious graphics issues. Obviously this machine was shipped without a human launching it because it happened right after launching. Everyone EXCEPT Stevie Wonder and Helen Keller could have seen it. I barely got the beta installed because of the problems but once installed I've not had one graphic issue since. Thars good news I guess?
So at the Apple Store while waiting the 2 hours I had iTunes open (files installed on computer no iCloud or internet) to see my playlists, but I was using DJay to test my crossfades on 350+ songs for an upcoming event.
I returned the unit through the online store the Genius Bar was not authorized to do anything except send it out for repairs r?? 17 days old out for repair??
When i returned it not only did I leave EVERYTHING installed, and an authorization agreeing to them using it this way (I have nothing to hide) I also sent the above letter and this statement:
"I'm including this info because if I was receiving this laptop I would want to know these things that were done and the particulars to help diagnose the battery issue.
Screen dimmed to 12 clicks WIFI turned off and unavailable Only two software titles running ever Pretty basic.
Hope this helps! Call me if you need clarification"
now apple claims safari caused it. HELLO? We never launched safari or mail or anything else only those two programs. So you can stop spreading the "all clear" signal Apple!
So you're suggesting that whatever issue you have with your copy (which may be defective) is related to what Apple and CR say was the issue in their testing? I'm not seeing the connection.
No. Just like my first sentence said "Fixed?? No it's not."
And it's not safari either because we never even used safari on this machine we had for 15 days. Never launched it once. Anyone stupid enough to think changing a setting in a browser will give me 7+ more hours of battery (we never even got 3 hours ever) then I have some snow to sell you. LOL.
I wrote that to show this laptop has serious battery issues and changing safari is NOT, I repeat NOT going to fix that. It may alter CR battery a few minutes but there's no way the battery not making it to 3 hours is going to get fix by clicking something on or off.
Fixed??? I think not! Let me share my experience that I went through:
friday 11:25 (the 6th): On Friday I got to the Apple Store at 11:30 with no appointment. I was sent there by AppleCare due to the extreme nature of my problem with my new laptop just 15 days old. I walked in with a fully charged battery. 100% AND stopped an employee so I'd have a witness.
I went because three nights in a row I took my laptop, bought my other half dinner at 6:45 then stayed to work till he got off at 10pm. So I worked from 7pm till 10. Three nights in a row my computer went from 100% to 1% by 10pm (actually 2 hours and 56 minutes to be exact).
The odd thing is on Monday night I got in line at 2:00pm for a movie premier. By 4:57pm after leaving the house 100% it was 1% and I barely was able to save my work to my iPhone. Then it shut down.
My first reaction? "I swear this was 100%. Maybe I'm on acid and trippin' ". LOL
In all 4 situactions I did not have WIFI or electricity so those options were not turned on. I also tried "graphics switching on" for 2 of the days, then "graphics switching off" the other two days. No improvement and it didn't get worse either.
So now it's Friday morning and at the apple store ... as I mentioned above. AppleCare sent me with no appointment. They made me wait for over two hours. That's ok. In this case that was a GOOD thing. I started logging in my battery drain using my iPhone. Check this out. Here's my battery while I waited: 11:35. 100% 11:48. 92% 12:01. 83% 12:08. 79% 12;27. 66% 12:46 52% 12:57. 46% 1:11. 38% 1:18. 33% 1:27. 27% 1:34. 21% 1:44. 15%
WOW! So here's the thing: although we installed all our software on this Machine we only ran 2 apps and both were optimized for the touch bar model
DJay 1.4.2 iTunes 12.5.4
That's it. This was to make sure we weren't running non optimized software. (I'm a beta tester for Apple so I went in to "beta tester" mode.). By limiting what I was using it helped control the experiment. One thing to note: the ONLY reason I installed my beta account was because with the golden master it shipped with I had serious graphics issues. Obviously this machine was shipped without a human launching it because it happened right after launching. Everyone EXCEPT Stevie Wonder and Helen Keller could have seen it. I barely got the beta installed because of the problems but once installed I've not had one graphic issue since. Thars good news I guess?
So at the Apple Store while waiting the 2 hours I had iTunes open (files installed on computer no iCloud or internet) to see my playlists, but I was using DJay to test my crossfades on 350+ songs for an upcoming event.
I returned the unit through the online store the Genius Bar was not authorized to do anything except send it out for repairs r?? 17 days old out for repair??
When i returned it not only did I leave EVERYTHING installed, and an authorization agreeing to them using it this way (I have nothing to hide) I also sent the above letter and this statement:
"I'm including this info because if I was receiving this laptop I would want to know these things that were done and the particulars to help diagnose the battery issue.
Screen dimmed to 12 clicks WIFI turned off and unavailable Only two software titles running ever Pretty basic.
Hope this helps! Call me if you need clarification"
now apple claims safari caused it. HELLO? We never launched safari or mail or anything else only those two programs. So you can stop spreading the "all clear" signal Apple!
So you're suggesting that whatever issue you have with your copy (which may be defective) is related to what Apple and CR say was the issue in their testing? I'm not seeing the connection.
No. Just like my first sentence said "Fixed?? No it's not."
And it's not safari either because we never even used safari on this machine we had for 15 days. Never launched it once. Anyone stupid enough to think changing a setting in a browser will give me 7+ more hours of battery (we never even got 3 hours ever) then I have some snow to sell you. LOL.
I wrote that to show this laptop has serious battery issues and changing safari is NOT, I repeat NOT going to fix that. It may alter CR battery a few minutes but there's no way the battery not making it to 3 hours is going to get fix by clicking something on or off.
So did Apple find the cause of what was wrong with your new MBP?
I believe what I see, and I see a consistent 10 hours battery daily use with my new MBP. This is the results that matter for average consumer I believe. As for CR, I don't care how they did their test, but I care that instead of contacting Apple for a questionable results they see, they went and published this to the media, resulting in a high profile media publication that is unfair to Apple.
I would guess that Consumer Reports did this knowingly. Their subscription base has likely declined as internet reviews of products are more immediate. Any chance to find something 'bad' about Apple really must be irresistible to them. I canceled my subscription many years ago, knowing that they are not the objective source they used to be.
Either way, if your watching video on Hulu, Netflix or DirecTV/Comcast or some other non-cached data, your not going to get the 10 hour battery life Apple is advertising...
This is absolutely stupid. And the condemnation has been posted on news site every where - including those such as CNN that do not accept reader comments.
I stopped trusting Consumer Reports a long time ago. The did a head-to-head test between a Mac and a couple of PCs. The Mac had Gigabyte Ethernet built into the mother board. As was typical, the PCs had to have an Ethernet card. Being the PC-heads that they were, it never occurred to them to look at the ports on the back of the machine. They just looked at the cards and decided that the Mac did not support networking. They also included this supposed lack of networking in the text of their condemnation and in their feature comparison grid. Naturally, they had a photo of the back of the machine and there was the RJ45 port, as plain as day.
... 2) Where Google created their own browser from the Chromium open source project - which is also essentially theirs - both Apple and Microsoft essentially cribbed theirs from Netscape and Firefox. (Particularly since the people who created Netscape left to form Firefox after Microsoft bought it.) When Microsoft deviated from the Netscape base in order to try to keep up with Firefox and Chrome they made a mess of things. Apple didn't even try to keep up in the browser wars so they just left it limited, without even trying to compete with Chrome and Firefox on functionality. That is why the very instant I read about the test that Consumer Reports was running, I instantly knew "bug in Safari, or something in the OS that interacts with Safari." And sure enough, even though Apple did their best to obfuscate by (less than truthfully) claiming that the test results were due to "hidden settings that never get used by consumers" they are indeed issuing a bug to fix the problem. And when I say obfuscate ... wow. Every single browser has that disable cache setting. Every. Single. One. And it is not hidden; it is right there in the browser settings. LOTS of people turn it off for various reasons. And you know what? Such as ... when you do "private browsing." That is right. Whenever you do "private browsing" IT ENABLES THAT SETTING. Let me repeat: PRIVATE BROWSING ENABLES THIS SETTING.
Wow... there is so much wrong in your entire long winded post.
First: Google Chrome is a direct descendent of Safari. For the longest time Google was using WebKit as the rendering engine for Chrome. WebKit is an open source project managed owned by Apple. WebKit has nothing to do with Netscape - It's based on/fork of the KHTML project which was mainly used for the Konqueror web browser in the KDE graphical environment on Linux. Google then forked WebKit to a new code base called blink. Opera also uses Blink as its rendering engine. Microsoft never purchased any assets from Netscape regarding the Navigator web browser. Those were passed to AOL before they were then open sourced, which became the "Phoenix" web browser, and then renamed Firefox. While Safari doesn't keep pace with all the features that Google Chrome offers, it's still rather current and usually supports the most important parts of the HTML5 standard, and is on a slower release cycle than Chrome and Firefox. If anyone "cribbed" a project, Google "cribbed" Chrome from Safari.
Second: Private browsing does not disable the browser cache. What private browsing does do is make temporary locations for your browser cache and cookies for a specific private browsing window. These temporary locations are then deleted immediately when the private browsing window is closed. Other web browsers follow the same principal. This is why private browsing is never a guarantee (and all the browsers warn you of this) of preventing someone to know your web browsing history if they have some type of access to the system while you are performing web browsing.
I've read neither the original post nor your rebuttal, but you get points simply for being able to decipher that huge body of unformatted text!
Apple's aptitude is typical of R&D engineers. They like to blame testers for using their software incorrectly. But this is not Jobs liked. R&D should implement software such that no matter which user uses the software it will always perform as specified and make the user happy.
That's not possible. You could produce a bicycle that works wonderfully when operated as intended. If a user chooses to let all the air out of the tires, however, it is unreasonable to expect that the bicycle would then operate just as well as it did with tires properly inflated. A user with that expectation will not be happy, and it's not the fault of the designer. The user is not operating the bicycle as specified, and should not expect the bicycle to nonetheless operate as specified. The user is just an idiot, and you can rest assured that's exactly what the late Steve Jobs would say about it, too.
3. According to APPLE, not CR, the problem is the result of a BUG, not the testing methodology per se. The testing method merely exposed it. How long would it have gone on unnoticed and unfixed if the test hadn't uncovered it? It may not have affected me or you, but obviously some people would suffer from it. This is a positive outcome.
Actually I think Apple blamed both the methodology and the bug. See their statement:
“We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug..."
Note that they use the word "also triggered" the obscure bug. "Also", not "subsequently" triggered the bug. Right or wrong, they seem to be saying both conditions are responsible for the results.
That's interesting. Maybe THAT's how Apple arrived at these unrealistic runtimes of "up to 10 hours" in their own testing: by visiting the same pre-programmed websites over and over again, allowing the usage of cache? That would be a typical "lab test" for marketing and not real life, where we visit a multitude of different websites that require constant loading of content.
I believe what I see, and I see a consistent 10 hours battery daily use with my new MBP. This is the results that matter for average consumer I believe. As for CR, I don't care how they did their test, but I care that instead of contacting Apple for a questionable results they see, they went and published this to the media, resulting in a high profile media publication that is unfair to Apple.
As one of the people who actually had these issues and returned the laptop because of it... YOUR post feels somewhat unfair... 10 hours consistently? Are you just letting your laptop sit in a corner?
I don't think anybody did anything wrong here -- neither Apple nor CR -- and both are handling it appropriately.
Agreed. Comparing one computer's caching strategy to another's isn't the goal. The goal is to test battery life under conditions as similar as possible among all models. Turning the cache off does, in fact, simulate the typical user's actions of going to different websites.
And while I'll admit I'm not very typical, I nearly always turn caching off in my browsers. For a variety of reasons.
This is absolutely stupid. And the condemnation has been posted on news site every where - including those such as CNN that do not accept reader comments.
I stopped trusting Consumer Reports a long time ago. The did a head-to-head test between a Mac and a couple of PCs. The Mac had Gigabyte Ethernet built into the mother board. As was typical, the PCs had to have an Ethernet card. Being the PC-heads that they were, it never occurred to them to look at the ports on the back of the machine. They just looked at the cards and decided that the Mac did not support networking. They also included this supposed lack of networking in the text of their condemnation and in their feature comparison grid. Naturally, they had a photo of the back of the machine and there was the RJ45 port, as plain as day.
I call bullshit. Show me this report.
Probably impossible as it was likely for the 1991 Mac Quadra 700 which was the first Mac to get built in Ethernet. Maybe CR has past issues online but they would likely be behind the paywall.
3. According to APPLE, not CR, the problem is the result of a BUG, not the testing methodology per se. The testing method merely exposed it. How long would it have gone on unnoticed and unfixed if the test hadn't uncovered it? It may not have affected me or you, but obviously some people would suffer from it. This is a positive outcome.
Actually I think Apple blamed both the methodology and the bug. See their statement:
“We learned that when testing battery life on Mac notebooks, Consumer Reports uses a hidden Safari setting for developing web sites which turns off the browser cache. This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug..."
Note that they use the word "also triggered" the obscure bug. "Also", not "subsequently" triggered the bug. Right or wrong, they seem to be saying both conditions are responsible for the results.
That's interesting. Maybe THAT's how Apple arrived at these unrealistic runtimes of "up to 10 hours" in their own testing: by visiting the same pre-programmed websites over and over again, allowing the usage of cache? That would be a typical "lab test" for marketing and not real life, where we visit a multitude of different websites that require constant loading of content.
Or you could simply read on the Apple site how they test and not make dumb assed comments:
Testing conducted by Apple in October 2016 using preproduction 2.6GHz quad-core Intel Core i7-based 15-inch MacBook Pro systems with a 256GB SSD and 16GB of RAM; preproduction 2.9GHz dual-core Intel Core i5-based 13-inch MacBook Pro systems with a 512GB SSD and 8GB of RAM (wireless web test and iTunes movie playback test); preproduction 2.9GHz dual-core Intel Core i5-based 13-inch MacBook Pro systems with a 256GB SSD and 8GB of RAM (standby test); and preproduction 2.0GHz dual-core Intel Core i5-based 13-inch MacBook Pro systems with a 256GB SSD and 8GB of RAM. The wireless web test measures battery life by wirelessly browsing 25 popular websites with display brightness set to 12 clicks from bottom or 75%. The iTunes movie playback test measures battery life by playing back HD 1080p content with display brightness set to 12 clicks from bottom or 75%. The standby test measures battery life by allowing a system, connected to a wireless network and signed in to an iCloud account, to enter standby mode with Safari and Mail applications launched and all system settings left at default. Battery life varies by use and configuration. See www.apple.com/batteries for more information.
I believe what I see, and I see a consistent 10 hours battery daily use with my new MBP. This is the results that matter for average consumer I believe. As for CR, I don't care how they did their test, but I care that instead of contacting Apple for a questionable results they see, they went and published this to the media, resulting in a high profile media publication that is unfair to Apple.
Publishing before contacting is consistent with CR's approach for the last 30 years, at least, that I've been reading them. Generally speaking, simply contacting a company producing a product with a defect doesn't accomplish much. Publishing results of a defect to a wide audience is much more likely to result in that correction happening. CR has done this same thing with cars, electronics, children's toys and virtually every other product category they test.
The name of the organization is Consumer Reports, after all, not Company Reports. They consider their primary responsibility to be consumers who are subscribing to their reports, not the companies producing the products they test.
Certainly questions the legitimacy of testing going forward for CR. If their experts failed to disable something so simple, are they really that much of experts at hand. The option was turned on in Safari. That's not something on by default. #Fail
Certainly questions the legitimacy of testing going forward for CR. If their experts failed to disable something so simple, are they really that much of experts at hand. The option was turned on in Safari. That's not something on by default. #Fail
Umm, it was actually the other way around, with CS disabling the browser cache just as they do with all the other laptops they battery test. But yes they changed the default setting which is typically "On".
Comments
And it's not safari either because we never even used safari on this machine we had for 15 days. Never launched it once. Anyone stupid enough to think changing a setting in a browser will give me 7+ more hours of battery (we never even got 3 hours ever) then I have some snow to sell you. LOL.
I wrote that to show this laptop has serious battery issues and changing safari is NOT, I repeat NOT going to fix that. It may alter CR battery a few minutes but there's no way the battery not making it to 3 hours is going to get fix by clicking something on or off.
I've read neither the original post nor your rebuttal, but you get points simply for being able to decipher that huge body of unformatted text!
Agreed. Comparing one computer's caching strategy to another's isn't the goal. The goal is to test battery life under conditions as similar as possible among all models. Turning the cache off does, in fact, simulate the typical user's actions of going to different websites.
And while I'll admit I'm not very typical, I nearly always turn caching off in my browsers. For a variety of reasons.
http://www.consumerreports.org/apple/apple-releases-fix-to-macbook-pros-in-response-to-consumer-reports-battery-test-results/
EDIT: Mentioned much earlier in the thread by @blastdoor ;
Sorry for repeating it.
Publishing before contacting is consistent with CR's approach for the last 30 years, at least, that I've been reading them. Generally speaking, simply contacting a company producing a product with a defect doesn't accomplish much. Publishing results of a defect to a wide audience is much more likely to result in that correction happening. CR has done this same thing with cars, electronics, children's toys and virtually every other product category they test.
The name of the organization is Consumer Reports, after all, not Company Reports. They consider their primary responsibility to be consumers who are subscribing to their reports, not the companies producing the products they test.