Inside Consumer Reports: Controversies surrounding the MacBook Pro and HomePod

Posted:
in General Discussion edited May 23
Enthusiasts have questioned Consumer Reports' assessments of Apple's products for many years. After looking at the test facilities, AppleInsider talked to Consumer Reports leaders about the various controversies of the past.

Consumer Reports' head-and-torso test for iPhone X


On May 17, AppleInsider paid a visit to the Consumer Reports offices and testing facility in Yonkers, N.Y., where we got an inside look at how the organization arrives at its conclusions about products.

When we sat down with the Consumer Reports decision-makers, one of the topics of discussion was the organization's past assessments of Apple products, including a series of controversies involving individual conclusions about the 2016 MacBook Pro and the HomePod in early 2017.

"Biased for consumers"

Everyone we spoke with at Consumer Reports adamantly denied that they have been unfair to Apple, arguing that their mission and methodology is entirely analytical and data-driven.

Accusations of bias or unfairness to certain companies, a Consumer Reports spokesperson noted, is "not a unique complaint." In fact, there were times when Consumer Reports has been accused of bias in the other direction -- of liking Apple and its products too much.

Consumer Reports' Apple products page


"We believe that we have certainly been fair when it comes to Apple products and we haven't treated them any differently than anyone else,'" said Jennifer Shecter, CR's Senior Director, Content Impact & Corporate Outreach. "We test more than 7,000 products and services every single year, and I deal with every company who has an issue and they say well, why do you like Apple so much more than you like my product.' or you have other companies who say "why don't you like [mine].'"

"But this not a question of preference," said Shecter. "This is a question of scientific evaluation and comparative testing. It is not about preference, that does not come into play into our testing."

"We treat every company and every product the same way. And it's been like that since 1936," Shecter added. "We deal with Apple products and Apple as a company the same way we deal with every other company. Empirically speaking, we evaluate every product as it comes, and every piece of feedback from the manufacturer as it comes, and that is our process. So we have had robust dialogue with Apple, throughout the processes, and we are responsive to feedback that we get from companies, too."

According to Consumer Reports' director of communications Barrie Rosen, the organization currently recommends Apple products in the categories of smartwatches, smartphones, tablets, streaming media devices, smart speakers, laptops, and all-in-one desktop computers -- just about every category in which Apple offers products.

As far as device usage goes, while touring Consumer Reports' offices, we did note that the desktop computers are PCs. However, we did see quite a few MacBooks on desks there, and when we met with the company personnel in a conference room, at least two had MacBooks in front of them. As for smartphones, we noticed about a 50/50 split between iPhones and Android devices in the offices.

Apple and Consumer Reports

As for Apple and Consumer Reports' relationship, they do have one, and judging by the team's comments about it, it's somewhere between cordial and cooperative.

"We have an ongoing dialogue with Apple," Shecter said. "Whenever we have a story that's coming out that we're going to call Apple out -- just like we would any other manufacturer -- we call them. And when they have questions, we answer them.

Consumer Reports' says that their direct dealings with Apple are with all sorts of different Apple personnel. Sometimes they talk to marketing people, sometimes product managers, and occasionally even engineers.

"It might start with a product manager," Maria Rerecich, director of the electronics testing team, said. "But we've often wind up in dialogue with the engineers on the product, because they have specific technical questions that they need to be asking and hearing our answers to. We'll go engineer-to-engineer at that point.

Shecter added that the companies' communications are usually done over the phone and that in-person meetings with Apple are less common, mostly because the two companies are headquartered on opposite coasts.

"As we do, we invite manufacturers to come and visit our facility. We're always eager to go to other manufacturers' facilities," she said.

"I think that when we've been in dialogue with them it's been good faith on both sides," said Shecter. "There have been times when we've had to be in constant contact if there's a story coming out, or if there's more stories coming out, but every time they have a question we answer their questions, truly, honestly, and responsively."

The MacBook Pro Story

Consumer Reports' 2016 MacBook Pro assessment


One of those times dialog was needed was following the release of Consumer Reports assessment of the 2016 MacBook Pro.

As AppleInsider reported in December of 2016, CR, upon the release of the newly refreshed MacBook Pro line, declined to recommend the new product, the first time it didn't recommend the MacBook Pro. Its decision was reversed the following month, following lengthy engagement with Apple and the release of a software fix.

The problem, it turned out, was that Consumer Reports had turned off caching prior to running the test, leading to the exposure of a battery bug that Apple later fixed. Apple said in public comments at the time that this resetting was "not a setting used by customers and does not reflect real-world usage."

The team at Consumer Reports gave us their side of what happened in that product assessment. Their engagement with Apple involved extensive communications, over the Christmas holiday that year.

"We run battery life tests on laptops," Maria Rerecich said. "And the way we run the battery life test is, we are doing a website cycle over Wi-Fi. We don't run those live over Wi-Fi, we have a server with the webpages stored on it so it's consistent and our testing has to be the same from time to time. So we don't want to encounter the vagaries of someone's website being down. So we have the webpages stored, we run those over wi-fi from our server, to the test machines, and we're basically cycling through webpages."

"We start with the computers fully charged, we set them all to the same brightness level, so again it's consistent across. So just let em run overnight, and then some, depending on [how long they last]," added Rerecich. "We log that recording, and basically once the laptop dies the watch stops, so we can tell exactly when it's petered out. We run that once, we record the results, and we run it a second time. we're looking for consistency within five or ten percent."

Macbook Pro 2016


"We set them up the same as much as possible, we go with default settings largely, but have turned the cache off, and the reason we turn the cache off is because we don't want the website to be sitting in cache on the machine because then it's not exercising properly," Rerecich continued.

"In the case of the MacBook Pro, that year [2016], when we run the two trials of the same test, we might get 14 hours and 14.1 hours. We may get 16 hours and then 16.3 hours. If it's more than that 5 percent variation, we'll run it a third time and see where it falls. The case of the MacBook, we got 10 hours, then we got 4 hours. We never see a variation like that. This was completely unusual. And the numbers told us that there's something going on. We were not able to evaluate that battery as a result, because what's the right number?' And we felt that based on that testing, there was something that was not explained, and we could not recommend the model as a result. We got in touch with Apple and we had a lot of back and forth with Apple."

"We got in touch with them beforehand and asked them for a comment," Shechter said. "Because we said we're seeing this issue, and it's gonna impact our ability to recommend the product. And what they told us for the story was they didn't know why that would be. They didn't understand why we were getting the results we were getting. So we asked them and they gave us a comment for the story. We published our story and then there had more questions about trying to figure out what the problem could be. So that was over Christmas, the year before [last], and we probably talked to them every single day of that Christmas holiday."

Eventually, CR gave their log of the test to Apple, who determined what had happened.

"Because you're doing it the way you are doing it, there's a bug being triggered in Safari," Shechter said they were told by Apple. "When you turn the cache off, this bug is being triggered. And while we respectfully disagree with you that this is a real-world test, or that this is the best indicator, they said you know, we want to make sure to them the recommendation seemed important, and so we went back and forth, and they decided to issue an update for the bug, and so they issued it, and we re-ran the tests, and it worked. So we were able to recommend it."

Shechter added that CR's policy is that they will revisit product assessments after they have denied a recommendation for a specific reason, once the issue behind that reason has been fixed. But they will not retest products that have already been recommended. If they did that, she said, "we'd be re-resting products all the time and we'd never even get to new products."

Consumer Reports sees the back and forth with Apple as an instance in which they and a manufacturer worked together to deliver a positive result for consumers.

The HomePod issue

Consumer Reports' first look at the Homepod in early 2018


Another rather contentious issue was Consumer Reports' assessment, this past February, of the HomePod speaker, "The Apple HomePod Sounds Good, but Other Smart Speakers Sound Better." Many, including AppleInsider, had taken issue with CR's determination that the HomePod fell below such competitors as the Sonos One and Google Home Max. Our editorial also questioned how Consumer Reports had reached some a determination in what appeared to be a single day.

CR responded to that editorial that week, concluding with an invitation to visit their facilities.

During that visit, the CR staffers further filled us in on what happened when they wrote about the HomePod. The assessment was, they said, a "first look" -- a collection of initial impressions of a product that Consumer Reports often publishes for certain high-profile products releases, including most iPhone and Galaxy devices. First look assessments will also sometimes take look at new, highly-touted features -- such as the Face ID that debuted in the iPhone X -- and also look at whether the product does certain things that it advertises doing.

HomePod 2018


"Because consumers will be very interested in a phone like that or a phone like the flagship Samsung phones or things of that nature, we will try to get results out the first day," Recevich said. "So we'll do this first look type of thing, with really big, important products that people are buzzing about. Because consumers want to know, with Consumer Reports- everyone else is talking about it, why aren't you talking about it? So that's why we did that for the HomePod -- not because it was Apple, but because these are the types of products that people are [interested in] -- when the Apple Watch first came out, we did that. But we do it for other products as well.

"Because the HomePod was sort of a hot product and we knew a lot of people were interested in what it was like and how it was. We did what was called a first look," Recevich continued. "In this case it wasn't a rented sample because we didn't have that available, but the first day we got them in, we did some testing, so that we could give an impression about it."

The first looks are usually presented as "early test results." Consumer Reports has published day-of-release first looks of the iPhone X, the iPhone 8 and iPhone 8 Plus, the 2016 Apple Watch and the Samsung Galaxy S 8. CR does not appear to have published first looks upon the releases of the Sonos One or Google Home Max.

Consumer Reports' early review of the iPhone X in 2017


"Typically on a first day, those first looks that we do, we'll get the product in, we'll rush it into the lab." said Rerecich. "We'll do the testing that we'll accomplish on the first day, and then we're gonna get together with editorial and talk about what we saw, what the results were, we'll ask questions, and we'll refine what that first day story is about."

"We did a listening test. We actually ran through our entire audio quality evaluation that day, and that was the basis of the first look, along with how it is to work with, and things like that." Amplified Rerecich. "It was just comparing the one column, which I believe is all we did on the first day. whereas the ranges that came out, combined everything else. For a speaker like that, the sound quality is the highest-weighted, it's the most important, and the other ones affected it. So, the ranking that came out in the ratings could have been slightly different from that first day, because that first day was just one component of the test.

Rerecich added that testing is not just about sound quality for speakers, but also about ease of use, features, and overall versatility. Following those evaluations, the full report is issued.

Jerry Beilinson, Consumer Reports' content development team leader and electronics editor for the Consumer Reports magazine, argued that the assessment, written by CR's Allen St. John, wasn't actually all that negative.

"Very good sound, far better than most other speakers, one of the best. The argument seems to be that [we said] it wasn't the best speaker that was ever released. It's recommended, it got a very good rating, and was with very similar rating numbers, edged out by two similarly high-end competitors."

A question of aesthetics

The Consumer Reports officials did point out one thing: Their testing process does not, formally, take into account the look and aesthetics of the product. And design, it should go without saying, is something historically very important to Apple.

"We don't care much about aesthetics," Mark Connelly, director of testing said during our tour of the facilities.

"We'll describe them sometimes," said Rerecich. "Maybe for a washing machine that's not so important, but for something like a speaker that you might have in your living room, you might care what it looks like. We don't rate it and say this one looks better than that one,' unless it's unusual in some way."

"Just to be clear, those are not part of the numerical rating, they don't get calculated as part of it," said Jerry Beilinson. "What Consumer Reports can do for you is give you comparisons you can't make yourself and judgments you can't easily make yourself. So when you're in the store, what is the performance in terms of speed of this laptop? Well I'm not going to look at it or read the spec but and necessarily know what is the battery life if a consumer is looking at a product they can tell whether they like the way it looks, they don't really need ConsumerReports or another outside authority to tell them whether they think the blue one or red one is better-looking."

"That's not a way in which a consumer might be lacking in information they need to make the purchase," said Beilinson. "What we're really trying to do, primarily, is fill consumers in with information that they need to be able to make good choices."

One side or another

We appreciate that Consumer Reports opened their doors to discuss testing methodology, making the entire process a little more transparent. Based on our commenters, AppleInsider readers and car aficionados have had a trust issue. Trust, once lost, is hard to regain.

In the case of the "antennagate," the public-facing information was about the signal attenuation when the phone was held, with the rest of the story behind a paywall. A paywall is any publication's right, as bills do have to get paid, but the information that was propagated wasn't the whole story -- as the group did recommend the iPhone in that paid report.

Steve Jobs during the Antennagate controversy


With the 2016 MacBook Pro, Apple's declarations on the matter, and Consumer Reports still differ. CR has clarified its stance on the matter to us now, and Apple has not beyond it being not a consumer setting -- so the whole story for users is somewhere in between. We still feel that this should have been addressed with Apple before publication of the initial test results, though.

Regarding the more recent HomePod and the audio comparison, AppleInsider still believes that the comparison the same day that the product was released wasn't handled well at all. Additionally, while an analysis in a vacuum of a product can be done rapidly, we do not believe that any kind of meaningful comparison between complex products like the HomePod and Google Home Max that Consumer Reports aims to do can be completed from start to publication in a matter of hours.

This all said, there are a few things we conclude from our visit to Consumer Reports. Having viewed their testing process and met with their team, we are confident that they do not harbor a purposeful anti-Apple agenda, nor is there any sort of conspiracy against Apple afoot behind the CR walls. Their complete testing and evaluation process is conducted with integrity and in good faith.

However, there may very well be something about CR's analytical, numbers-driven process that clashes with the design-heavy Apple ethos, and makes their conclusions about Apple products different from those of more traditional reviewers. Even so, this hasn't stopped them from recommending most of Apple's lineup.
Alex1Nlolliver
«13

Comments

  • Reply 1 of 42
    dws-2dws-2 Posts: 173member
    The trouble with Consumer Reports is that they believe adding up all the numbers for a bunch of discrete tests actually tells you how good a product is. They don't have any Apple bias, except that Apple products tend to be more about the experience than the specifications.
    cornchipyojimbo007racerhomie3radarthekatjony0lolliverglee217
  • Reply 2 of 42
    saldogsaldog Posts: 35member
    Doesn't HomePod improve its sound over time? How could anyone evaluate it out of the box and in one sitting?
    lolliverchasm
  • Reply 3 of 42
    Yeah, well, after Anntenagate and following a painful experience with a BD weed trimmer that had a five star rating AND a serious engineering flaw, I dumped Consumer Reports forever and saved the subscription fee. The flaw? That was the fact the belt drive would melt the plastic drive pulley after about 2 hours total run time. Is the pulley replaceable? No, that’s part of the motor assembly. $80 and you install if the warranty is out or voided. I voided mine because I REALLY wanted to know what was going on and not comfortable with taking it in for warranty service every othe time I used it. FIVE STARS! Bye-bye CR…

    Alex1Nradarthekatlolliver
  • Reply 4 of 42
    tewhatewha Posts: 5member
    dws-2 said:
    The trouble with Consumer Reports is that they believe adding up all the numbers for a bunch of discrete tests actually tells you how good a product is. They don't have any Apple bias, except that Apple products tend to be more about the experience than the specifications.
    Consumer Reports seems to think transparency in how bad their testing is means that scientifically accurate tests of real world scenarios are not required. Who using debugging tools to disable cache and calls that a real world scenario? Great on them for admitting they did that (eventually), but it's still not a real scenario.
    tmaydewmeAlex1Nradarthekatjony0lolliverchasm
  • Reply 5 of 42
    wizard69wizard69 Posts: 12,424member
    dws-2 said:
    The trouble with Consumer Reports is that they believe adding up all the numbers for a bunch of discrete tests actually tells you how good a product is. They don't have any Apple bias, except that Apple products tend to be more about the experience than the specifications.
    Actually the problem here is with forum members that are butt hurt over one data point.   

    Say you are about to buy a new truck and start asking people how they did with their recent purchase.   Do you take the comments of one person as an absolute reflection on the dealer and the car manufacture?   Most people wouldnt.  

    Some of consumer reports more useful data actually comes from longterm studies such as repair frequencies on cars and msjor appliances.  The reality is you can only uncover so much in product testing of new stuff.  The real value in consummer testing comes from the long term.   A csr that suffers from major breakdowns well before 100,000 miles sjould cause people to reconsider. 

    In the end all Consumer Reports does is offer a valid data point.   How one uses that info defines its value.    
    tallest skilmuthuk_vanalingamwelshdogmaciekskontaktviclauyycgatorguyAlex1Njony0cjaer
  • Reply 6 of 42
    nunzynunzy Posts: 320member
    They are spec whores pure and simple.

    They downplay the stuff that really matters, like industrial design and ease-of-use. Who really cares how good a speaker sounds if it is hard to set up and looks ugly?
    radarthekatlolliver
  • Reply 7 of 42
    jdgazjdgaz Posts: 264member
    I am going to bet that they never listened to any of those speakers off axis. Direct radiators will be awful when listened to off axis. HomePod is much more forgiving when listening from just about anywhere in the room it is in. CR misses one of the key benefits by not knowing how most people at home listen to music. We move around.
    viclauyycAlex1Nradarthekatjony0lolliverchasm
  • Reply 8 of 42
    tallest skiltallest skil Posts: 43,119member
    nunzy said:
    Who really cares how good a speaker sounds if it is hard to set up and looks ugly?
    Uh... I do.
    welshdogrevenantAlex1Njony0GG1JosephAUpvonkbeowulfschmidtcjaer
  • Reply 9 of 42
    22july201322july2013 Posts: 163member
    Comparing two products which run two different operating systems (and provide two different sets of online services) is like comparing a car to a motorcycle.
  • Reply 10 of 42
    nunzynunzy Posts: 320member
    nunzy said:
    Who really cares how good a speaker sounds if it is hard to set up and looks ugly?
    Uh... I do.
    But most people don't care if a speaker sounds 1 percent better if you need to put them on special stands sticking out into the room. The Apple speakers over the years have never been the best sounding, but they  sound good enough for most people.

    Just like the AirPods. Nobody claims they are the best sounding headphones on the market. But they are good enough, and very easy to set up.
    radarthekatlolliver
  • Reply 11 of 42
    glhglh Posts: 4member
    I just bought a 2017 Nissan Maxima, even though the CR review said the ride made some of their testers feel sick. So, someone eats a bad hamburger for lunch, gets in the car, goes for a ride and feels sick. Others hear his or her complaint and say they feel a bit sick too. So much for objectivity -- actually the ride is smooth, much smoother than the 2017 Camry (rated much higher by CR) we rented. I really think CR's reviews are at the best highly subjective and wonder why so many people pay attention to them. They claim to be unbiased, but I can't accept that -- their ratings depend on the whims of the testers, at least when they're not actually doing scientific testing with instruments (and even then there's room for subjectivity). As far as HomePod testing goes, surely this is 95% subjective. If you get one and really like it, why should you care what some CR tester (who may have had a fight with a colleague an hour before) thinks?
    revenantlolliver
  • Reply 12 of 42
    tallest skiltallest skil Posts: 43,119member
    nunzy said:
    But most people don't care if a speaker sounds 1 percent better
    Very true.
    good enough for most people
    A disheartening tagline that Apple has embraced as of late.
    Alex1Nnunzyhammeroftruth
  • Reply 13 of 42
    Consumer Reports have given distorted reviews of Apple products for, literally, decades.  Back when PC's had internal jacks for sound cards, graphic cards, etc., Mac computers were regularly downgraded for not having this internal flexibility, (which virtually no one ever used or needed to use).  When PC's were running those awful early  and mid-life versions of Windows, even after the Mac switched to OSX, and there was a noticeable and significant difference in usability of the Mac compared to any PC, CR religiously ignored the operating system differences and compared only purely mechanical differences like screen and sound quality.  Only when PC's became total magnets for scam software and viruses, and Macs were virtually free of these onslaughts, did CR mention the Mac software advantage.  Usability has never been an issue or a concern for CR.  I suspect the CR engineers consider it to be a flaw, diminishing the importance of being a hardware/lousy software geek.
    These narrow minded engineering points of view, ignoring the user experience, has been a continuing attitude at CR, often blinding their evaluators and evaluations of products that real people have to use on an ongoing basis.
    Alex1Nradarthekatlolliver
  • Reply 14 of 42
    dewmedewme Posts: 1,330member
    tewha said:
    dws-2 said:
    The trouble with Consumer Reports is that they believe adding up all the numbers for a bunch of discrete tests actually tells you how good a product is. They don't have any Apple bias, except that Apple products tend to be more about the experience than the specifications.
    Consumer Reports seems to think transparency in how bad their testing is means that scientifically accurate tests of real world scenarios are not required. Who using debugging tools to disable cache and calls that a real world scenario? Great on them for admitting they did that (eventually), but it's still not a real scenario.
    Totally agree on your point about turning off the cache that is designed to be on all of the time being wrong. Caching in general is the single most effective and universal performance technique that’s been used in computing over the past 50 years. It’s like removing the rubber tires from a car’s wheels before running a slalom test. Plain stupid. 
    Alex1Nradarthekat
  • Reply 15 of 42
    StrangeDaysStrangeDays Posts: 4,769member
    nunzy said:
    But most people don't care if a speaker sounds 1 percent better
    Very true.
    good enough for most people
    A disheartening tagline that Apple has embraced as of late.
    As of late? I think you mean as of ever. “The computer for the rest of us” has an original Macintosh slogan. Even the Apple II was shown in family living rooms rooms in ads. They’ve always been designed and marketed for normals, and not spec junkies or neckbeards. 
    Alex1Nradarthekatnunzy
  • Reply 16 of 42
    joefratjoefrat Posts: 7member
    I never believed they are harboring some grudge.  I am sure they believe what they publish.  What bothers me is the way they (along with many media outlets) dramatize their findings.  It appears they are trying to drum up interest in themselves by releasing a dramatic headline.  That is what I do not recall happening in the Consumer Reports of the past.  Just like as was mentioned in this article: the dramatic headline brings the reader to a paywall to find out the reality of the details.  That, to me is not what I would expect from the impartial Consumer Reports of the past.
    Alex1Nradarthekat
  • Reply 17 of 42
    jimh2jimh2 Posts: 30member
    Consumer Reports has been overtaken by the Internet. Nobody reads or cares about their reviews except other review sites lazily recapitulating what CR writes. As for not caring about the design of a product I do agree with them. They are testing functionality and performance; visual appeal should not factor into the results, unless all other things being equal it can be the differentiator. In the end none of these reviews mean anything. People buy what they want.
    Alex1N
  • Reply 18 of 42
    I've been a CR online subscriber for years. I often don't agree with their assessments, but I think it's important that someone do this sort of independent testing and they are the only option. So, I support them with my subscription. That said, I wouldn't buy or not buy anything based exclusivity on a CR test report. It's a data point. One that I would rate much higher than a manufacturer's marketing blurb or your typical Amazon review, but still just one data point. To me it's worth the minimal cost of the CR subscription.
    SoliAlex1NGG1glee217
  • Reply 19 of 42
    viclauyycviclauyyc Posts: 251member
    nunzy said:
    Who really cares how good a speaker sounds if it is hard to set up and looks ugly?
    Uh... I do.
    What speaker you use?
    i have an old pair of B&W Matrix 800. It was something 30 years ago. Now it is almost butt ugly 
    Alex1Nnunzy
  • Reply 20 of 42
    cornchipcornchip Posts: 1,048member
    nunzy said:
    But most people don't care if a speaker sounds 1 percent better
    Very true.
    good enough for most people
    A disheartening tagline that Apple has embraced as of late.
    Hasn’t that Ben Apple’s whole bag all along?
    nunzy
Sign In or Register to comment.