Consumer Reports' dismissal of HomePod a familiar tale to Apple fans [u]

Posted:
in iPod + iTunes + AppleTV edited February 14
Testing stalwart Consumer Reports declared this week that Apple's HomePod falls behind Google's Home Max -- and even the Sonos One -- in terms of audio quality. To anyone who knows the publication's controversial history with Apple products, that conclusion is utterly unsurprising.




In Friday's screed, the same day as the product was released, Consumer Reports, after less than a day of use and comparison, declared the Sonos One, and the Google Home Max the champions for sound quality. That's noteworthy, because virtually every other major test conducted since the launch of HomePod, including AppleInsider's own comparisons, found that Apple's product offers superior sound.

Don't just take our word for it (twice), though. Engadget on Tuesday agreed with the prevailing wisdom on it, as did USA Today, The Verge, What Hi-Fi, a very particular Redditor, and a pile of other venues.

Consumer Reports is, of course, entitled to its opinion -- sound quality assessments are highly subjective.

But two factors suggest that the report isn't relevant, nor of any particular value. Primarily, Consumer Reports has a long history of berating Apple products for issues that don't seem to actually manifest in any real-world environment. Second, their HomePod comparison has come to a conclusion that only they seem to hold.

Antennagate

Back in 2010, Consumer Reports used its public-facing webpage to flog issues it had with the iPhone 4 antenna. But it required a subscription to the site to view the entire comparative analysis, where it ranked the iPhone 4 above all other smartphones of the day, saying that the antenna issue was completely solved by applying piece of transparent tape over the antenna line in the casing, using a protective case, or not squeezing it with a "death grip" while making a phone call.

That fired up the internet as a whole, who saw only the public-facing post decrying the antenna -- yet the same complainants were not widely exposed to the full report stashed behind a paywall that said that the problems were incredibly minor and the iPhone 4 was at the top of the heap.




As a result of the growing controversy, Apple uncharacteristically made a public statement, and provided free cases for early adopters, in an attempt to put the issue to bed.
Gripping any mobile phone will result in some attenuation of its antenna performance, with certain places being worse than others depending on the placement of the antennas. This is a fact of life for every wireless phone. If you ever experience this on your iPhone 4, avoid gripping it in the lower left corner in a way that covers both sides of the black strip in the metal band, or simply use one of many available cases.
The laws of physics are inescapable, and every smartphone even now can be shielded by human flesh when pressed up against an antenna gap

And yet, somehow, this only applied to the iPhone 4, and not the other devices on the list according to the venue -- even though at the same time literally every other phone on their evaluation was seeing the same thing.

Consumer Reports didn't bother to say anything about it afterward.

The 2016 MacBook Pro

Apple released the MacBook Pro redesign in the fall of 2016, and there was much to be said about it. As it always does, Consumer Reports got their hands on one shortly after the machines shipped in late October and early November.

After about a month and a half of testing, in a post on their website, the publication said that while the new machines earned high marks in display quality and performance, they were found lacking in terms of battery life. Specifically, the battery performance of models tested "varied dramatically" during trials, and was as low as four hours, with the computers that were not under serious load.

However, it was discovered that, during their testing, Consumer Reports toggled a hidden developer setting, which triggered an obscure bug. While the bug's presence was Apple's responsibility, tooling around in the developer's settings for Safari isn't a "real world" test, contrary to what publication claims.

"This is not a setting used by customers and does not reflect real-world usage. Their use of this developer setting also triggered an obscure and intermittent bug reloading icons which created inconsistent results in their lab," Apple said. "After we asked Consumer Reports to run the same test using normal user settings, they told us their MacBook Pro systems consistently delivered the expected battery life."

Following the bug fix, and Consumer Reports not messing around with developer settings, the product testers reinstated their recommendation for the machine -- an improvement over how they handled the iPhone 4 debacle.

Unlike its declaration about the HomePod Monday just three days after release, Consumer Reports waited until its month-long testing was fully complete on the MacBook Pro before it published the results.

The iPhone X

The publication also disclosed its testing on the iPhone X and a bevy of other smartphones on Dec. 5 of last year, a month after the iPhone X became available. That report found the publication recommending the Galaxy S8 above the iPhone 8 Plus, with the iPhone X in a very close third spot.

The testing was dramatic, with the phones taking major screen damage in a steel tumbling device after 50 rotations. We're not sure how "real world" this is, as the vast majority of iPhone drops are from approximately three feet to concrete, and only a few times in a product's life.

The iPhone X tumbler
The iPhone X tumbler


However, again, Consumer Reports didn't talk about early results until they had been beating on the phones in question for a month -- more than 10 times the length of time it had before it judged the HomePod.

Review challenges

Consumer Reports has every legal right to share its opinions, the same as AppleInsider or any other venue. Examining a product in a review is, by definition, a subjective process, with audio product reviews producing the widest gap of opinion.

If you're looking for an "objective review," or similar phrasing, you aren't looking for a review at all. Instead, find a news item dispassionately discussing the technical specifications of a product. By definition, a review isn't that, and is chock-full of opinions about a product.

Perhaps Consumer Reports felt like they had collected all the information they needed after just a few days. That's possible, and with a product evaluated in a vacuum, and not compared to much else, a review can be done in less than that timeframe. However, this is contrary to every other consumer electronics comparison that the organization has ever done.

Damning with faint praise

It would have been better had Consumer Reports waited until their comparative testing was done to scream loudly that one smart speaker sounds better than the next. But, instead, this nearly century-old publication decided to say very little about how they reached their conclusions. Instead, they sensationally disclosed their findings, some of which flew contrary to everyone else's. And they did so just days after a product release, and yet they still give themselves an out if their exhaustive testing proved otherwise.

Why they took this atypical path to publication, we don't know, and can only guess, since they have yet to respond to questions about the release.

Right now, we don't know anything about the testing methodology. We don't know why Consumer Reports decided what it did, despite their press release vividly declaring that Google's offering is better.

I am not here to defend the HomePod, and Apple neither requires nor desires my assistance -- and that's how it should be. I don't have a HomePod, nor do I have a Google Home Max, so I can't speak to the accuracy of the Consumer Reports declaration.

But Monday's report, how it was disclosed, and a lack of a supporting narrative for the opinion, is not a good look for Consumer Reports, given their history with Apple products. The release is contrary to their own stated mission of "rigorously testing products" before advising the public.

Worse yet, the early announcement stands in violation of what I think is their most profound mission -- they have failed to be a publication capable of "fearlessly investigating where markets have failed," and look to have become part of the problem instead.

Update: Consumer Reports published a rebuttal on Tuesday evening. They noted in the rebuttal that they published their first look at the HomePod on Friday, and not Monday, and differed with some of our interpretations of the examination -- but we stand by our stance on the matter.

AppleInsider has asked some follow-up questions, specifically about how Consumer Reports came to their conclusion about the HomePod and their interpretation of the MacBook Pro testing regimen versus what Apple had to say about the dialog, and will update accordingly.
jahbladeargonautscampercomJWSCalbegarclolliver
«13456

Comments

  • Reply 1 of 106
    Consumer Reports might not sell advertising space, but they know to sell magazines all they have to do is trash Apple. 
    lordjohnwhorfinjahbladelostkiwimagman1979netmagealbegarcRobPalmer9racerhomie3lolliverjony0
  • Reply 2 of 106
    lkrupplkrupp Posts: 5,653member
    Unfortunately a significant segment of the population take CR’s reviews as gospel and make buying decisions based solely on what CR says about a product. I wonder, though, if CR’s negative attitude toward anything Apple has actually affected sales.
    edited February 13 tmayargonautmagman1979netmagelolliverjony0watto_cobra
  • Reply 3 of 106
    I trust Consumer Reports. In general they have always given Apple positive reviews. 
    tokyojimu
  • Reply 4 of 106
    jdgazjdgaz Posts: 220member
    More fake news. Sending letter to CR and cancelling subscription. Had enough of their not so professional evaluations.
    magman1979netmagejony0watto_cobra
  • Reply 5 of 106
    BluntBlunt Posts: 158member
    Bunch of clowns as a don't live in the USA i don't no how important they are. is it a big deal?
    macpluspluswatto_cobra
  • Reply 6 of 106
    I found their reviews for cars to be utterly useless decades ago when I was buying my first car and haven’t bothered with them ever since.
    edited February 13 jahblademacplusplusmagman1979netmageericthehalfbeealbegarclolliverMplsPjony0watto_cobra
  • Reply 7 of 106
    neilmneilm Posts: 515member
    I'm not here to defend the HomePod, which I've never even seen, much less heard.

    But at least a decade ago I came to the conclusion that, on products I knew anything about, I almost never agreed with CR's recommendations. Whatever it is they care about, it's not what I care about. On that basis, why take their advice on any other products? And why continue to subscribe to CR? So I don't.
    edited February 13 SpamSandwichchristopher126pscooter63jahbladeargonautmagman1979netmagealbegarclolliverMplsP
  • Reply 8 of 106
    jdgaz said:
    More fake news. Sending letter to CR and cancelling subscription. Had enough of their not so professional evaluations.
    And that’s your comment? That it’s “fake news”? Explain. 
    christopher126argonautbonobob
  • Reply 9 of 106
    Consumer Reports works with advertising lawyers, often providing information used by them to bring class actions. I’ve never understood why anyone relies on their opinions.
    watto_cobra
  • Reply 10 of 106
    f1ferrari said:
    Consumer Reports might not sell advertising space, but they know to sell magazines all they have to do is trash Apple. 
    And I suppose it’s possible that, even though they don’t take advertising dollars, they accept money which can skew their review positively or negatively. 

    I gave up on CR years ago when I noticed that their “CR Best Value” tag (or whatever it says) was always simply the cheapest of what was being reviewed, even if its scores were bad.

    It also used to bug me when they would rank the ‘value’ of a Mac lower (due to its higher cost compared to the PCs in the review) but disregard all the software that was included with the purchase of a Mac and that worked well together (iPhoto, iTunes, iMovie, iDVD, iWeb, etc). Finding feature equivalent on the PC side turned out to be several hundreds of dollars and they usually didn’t integrate well.
    jahblademagman1979lolliverjony0watto_cobra
  • Reply 11 of 106
    zoetmbzoetmb Posts: 2,234member
    CR does two types of reviews:  one uses objective measurements and another uses a consumer panel.   I don't know what they did in this case because I no longer read it, but consumer panels are absolutely worthless for evaluation of audio and video because most consumers have absolutely no idea what constitutes good audio (or picture) - they just know what they think they like.   Consumers will choose a speaker that appears to have more bass and high end over one that doesn't, even if that sound is inaccurate.  They'll also choose whichever system is louder, even if it's as little as 1db difference.   And they'll choose an over-saturated high contrast picture over a properly calibrated one.   That's why retailers set their TV displays to "store blast" mode.  

    Back in the 1970's, CR rated the AMC Gremlin as the best subcompact made.   As a result, I bought one.   Everything that could possibly go wrong in a car (valves, rings, wheel bearings, oil leaks, electrical shorts, etc.) went wrong in that car.   The only thing that always worked fine was the air conditioner and the radiator - the car never overheated even in the desert.   Even the metal that held the seatbacks up wore away and I had to stick a rod in the seat to keep the back from falling down.   CR later apologized, but the damage was done.   I've never trusted them since.  

    The one thing CR is good for is after a product has been out for some years is looking at the repair reports.   Other than that, I find their ratings usually useless - either because their testing isn't valid in the real world, because they only test a limited number of models or because their consumer panel testing is too subjective.   And really - you don't need CR to evaluate audio quality:  just go into a store and listen.  


    edited February 13 tokyojimulollivergilly33watto_cobra
  • Reply 12 of 106
    Rene Ritchie of iMore is also trashing Consumer Reports today on Twitter. I agree with him on this.
    magman1979netmageRobPalmer9lolliverwatto_cobra
  • Reply 13 of 106
    I don't know what it is with CR...had a subscription 25 years ago. it's ok for reviews of vacuum cleaners, but they are too enamored with Windows and Costco-esque type cheap, crappy products.

    They don't get Apple's ethos.

    Too bad.

    Best
    randominternetpersonbattiato1981argonautmagman1979macky the mackywatto_cobra
  • Reply 14 of 106
    lkrupp said:
    Unfortunately a significant segment of the population take CR’s reviews as gospel and make buying decisions based solely on what CR says about a product. I wonder, though, if CR’s negative attitude toward anything Apple has actually affected sales.
    Oh yes, after they rated the iPhone 4 as "not recommended" sales practically stopped. Wait, no they didn't. Nobody gave a rat's behind, because they know CR is completely useless at testing electronics, audio equipment and cars. Seriously, how do you give Tesla a highest rating ever mark one year and an unsatisfactory do not buy the next? They should stick to reviewing laundry detergent and fridges. And I'm not sure I even trust them for that.
    magman1979netmagelollivermacky the mackybrucemcwatto_cobra
  • Reply 15 of 106
    fallenjtfallenjt Posts: 3,669member
    I rather go on Youtube to watch the reviews from the real recording artist or musician than those mediocre CR. That's why I cancelled my subscriptions years ago..
    magman1979albegarclollivergilly33watto_cobra
  • Reply 16 of 106
    Consumer Reports should stick to what it knows. Exactly what that is, I couldn't tell you. Aside from their automobile review fiascos, they also had extremely questionable advice on bicycles, enough that one of the major US bike magazines spent several hilarious pages lampooning them with ridiculous "advice" for cyclists.
    netmagelolliverwatto_cobra
  • Reply 17 of 106
    All you have to do is follow the money.  From the Consumer Reports site when you click a shopping link (surprisingly it’s Amazon)...”Please note that Consumer Reports collects fees from both eBay Commerce Network and Amazon for referring users.”

    Should demonstrate why the Amazon products will almost always top other brands.
    zroger73lolliverwatto_cobra
  • Reply 18 of 106
    That was a very even-handed response to CR's suspect assertions. Well done. :)
    pscooter63netmagenhugheslolliverwatto_cobra
  • Reply 19 of 106
    SoliSoli Posts: 6,197member
    I trust Consumer Reports. In general they have always given Apple positive reviews. 
    I don't trust CR, but their results mostly match my experience with the HomePod. As even AI's video comparison shows, if you have a Google Home Max as your sole speaker system under your TV facing your couch, it'll be a better result than the HomePod. They didn't test two Sonos Ones acting in unison, but the results of the single Sonos One were close enough that I have to assume that the same setup would be better than a single HomePod or Google Home Max, especially since you can separate the sound better. There are reviews that have the Sonos Play 3 and Play 5 as being better single speaker options for directional setups, and, of course, Apple fanatics are first make claims that no one uses any Alexa of Google Now features except for playing music (denial, plus it allows them to negate that people can easily connect their ), which is usually followed by comments that they'll make it a better digital personal assistant soon (sounds like how Android fanatics talk about Android on this forum), which all ignores that so many said that Apple would've made a home-based digital personal assistant years ago if there was ever a market for it and that they'd never have something in their home that could always be listening (despite having mics, cameras, and internet-connected devices in pretty much every facet of their life).
    edited February 13 larrya
  • Reply 20 of 106
    Mike WuertheleMike Wuerthele Posts: 1,929administrator
    Soli said:
    I trust Consumer Reports. In general they have always given Apple positive reviews. 
    I don't trust CR, but their results mostly match my experience with the HomePod. As even AI's video comparison shows, if you have a Google Home Max as your sole speaker system under your TV facing your couch, it'll be a better result than the HomePod. They didn't test two Sonos Ones acting in unison, but the results of the single Sonos One were close enough that I have to assume that the same setup would be better than a single HomePod or Google Home Max, especially since you can separate the sound better. There are reviews that have the Sonos Play 3 and Play 5 as being better single speaker options for directional setups, and, of course, Apple fanatics are first make claims that no one uses any Alexa of Google Now features except for playing music (denial, plus it allows them to negate that people can easily connect their ), which is usually followed by comments that they'll make it a better digital personal assistant soon (sounds like how Android fanatics talk about Android on this forum), which all ignores that so many said that Apple would've made a home-based digital personal assistant years ago if there was ever a market for it and that they'd never have something in their home that could always be listening (despite having mics, cameras, and internet-connected devices in pretty much every facet of their life).
    That may be the case, and I see where it might be accurate. But, they went out of their way after three whole days of testing to declare a winner. Not only that, they didn't talk about the whys and the hows -- just that it wasn't Apple.

    That's my issue. I'm fine with them declaring whatever the winner. But, that's not enough -- and leaving it at that breaks the group's stated goal of exhaustive testing.
    edited February 13 Bluntmagman1979lolliverwatto_cobra
Sign In or Register to comment.