Inside Consumer Reports: Controversies surrounding the MacBook Pro and HomePod

2

Comments

  • Reply 21 of 42
    SoliSoli Posts: 10,035member
    I don't see CR being biased for or against Apple, but I do question their methodology. I much prefer the in-depth testing of other companies, and even individual testers for all my computer-based products. CR hasn't been a source I seek out for decades and I don't see that changing anytime soon.
    Alex1Nradarthekat
  • Reply 22 of 42
    sflocalsflocal Posts: 6,092member
    nunzy said:
    Who really cares how good a speaker sounds if it is hard to set up and looks ugly?
    Uh... I do.
    I would think (hope) Consumer Reports would report that the speaker sounds great, but it is a royal pain in the backside to set up.  That would be enough for everyone to make their choice, or at the very least be minimally surprised.
    Alex1Nnunzy
  • Reply 23 of 42
    9secondkox29secondkox2 Posts: 2,666member
    That’s a lot of excuses and doublespeak. No thanks CR. 
  • Reply 24 of 42
    radarthekatradarthekat Posts: 3,842moderator
    jimh2 said:
    Consumer Reports has been overtaken by the Internet. Nobody reads or cares about their reviews except other review sites lazily recapitulating what CR writes. As for not caring about the design of a product I do agree with them. They are testing functionality and performance; visual appeal should not factor into the results, unless all other things being equal it can be the differentiator. In the end none of these reviews mean anything. People buy what they want.
    All other things don’t need to be equal for UX to be a differentiator.  Anyone with the skill and knowledge can purchase components and build a Windows PC that beats an iMac, for example, on specs, and likely for lower cost.  But it may not be a better machine for a normal person to use.  The Mac OS gives an advantage in usability for most people that outweighs higher internal specs and lower cost.
    edited May 2018 dewmelkrupp
  • Reply 25 of 42
    flippyschflippysch Posts: 29member
    My skepticism of the validity of Consumer Reports review began in the early 1990's when I purchased a Chevy Corsica that Consumer Reports gave a solid black, worst, rating for most categories. I had it for 10 years and over 130,000 miles, with the only thing ever going wrong was with the power window motor dying in year 10. The CR rating for my product was way off the mark.

    However, it was not until I followed the early 2000's Consumer Report review of the Maytag Neptune that made me forever skeptical. It had the highest rating of washing machines, so I bought it. I rank it as the worst purchase I have ever made. We saw the Maytag repairman more times in our house than on television commercials. I happened to have an autographed picture of Gordon Jump as the Maytag repairman and use to pull it out and show the real repairmen when they came over to try and repair the mold producing, multiple failed logic board, beast -- Consumer Reports top selection.      
  • Reply 26 of 42
    Consumer Reports have given distorted reviews of Apple products for, literally, decades.  Back when PC's had internal jacks for sound cards, graphic cards, etc., Mac computers were regularly downgraded for not having this internal flexibility, (which virtually no one ever used or needed to use).  When PC's were running those awful early  and mid-life versions of Windows, even after the Mac switched to OSX, and there was a noticeable and significant difference in usability of the Mac compared to any PC, CR religiously ignored the operating system differences and compared only purely mechanical differences like screen and sound quality.  Only when PC's became total magnets for scam software and viruses, and Macs were virtually free of these onslaughts, did CR mention the Mac software advantage.  Usability has never been an issue or a concern for CR.  I suspect the CR engineers consider it to be a flaw, diminishing the importance of being a hardware/lousy software geek.
    These narrow minded engineering points of view, ignoring the user experience, has been a continuing attitude at CR, often blinding their evaluators and evaluations of products that real people have to use on an ongoing basis.
    A couple of things along these lines have turned me off of CR for a long time.  Years ago I remember them specifically calling out the cost as a negative on a Mac, but ignored that it came with a bunch of software (iLife Suite, for example) that the PCs they were also reviewing lacked.  Around the same time I read another article (not on CR) that did it’s best to match the software in iLife on the PC side with third-party solutions and the cost was a few hundred dollars.  So, to ignore that but complain about cost rubbed me the wrong way.

    I also don’t understand how many times the CR Best Buy (or whatever term they use) doesn’t necessarily have the best or even good rating but DOES have the lowest or close to lowest cost.  Do people really subscribe to CR to find out what refrigerator costs the least?  I certainly didn’t.  I wanted to know what rated well.  If they had ranked them by the ratings and then figured out which units were really good but also were a good value and highlighted those it would be a different case entirely. Meaning, if one refrigerator ranked the hightest in everything but was twice as expensive as the one just below it, call out the second one.  But, no, they call out the one that ranked 9th but was really inexpensive.  Huh?

    The only thing we really look to CR for anymore is the report you can purchase to see what the lowest possible price you should be able to negotiate to on a specific model of car is.  We’ve found that to come in very handy.
  • Reply 27 of 42
    AppleZuluAppleZulu Posts: 1,989member
    I was really hoping that the conversation about the HomePod review would have taken CR to task for using a methodology that goes a long way to defeat a key aspect of the device. HomePods analyze the acoustics of the room and modify the signal coming out of the speakers to cancel out undesired effects of the room and they also use acoustically reflective surfaces in the room to actually enhance the sound. CR tests speakers in a room designed to deaden surfaces by absorbing sound waves that would otherwise be reflected. That’s possibly a “fair” way to compare standard speakers to one another, though still not a real-world test. Nobody lives in a place with acoustical tiles all over the room. For the HomePod, that room will defeat precisely the thing that sets the HomePod apart, and that’s not a fair way to assess the device. Compare a HomePod to other speakers in, well, a home, and it will shine where others are diminished.

    If you tested a Rickenbacker 12-string electric guitar by playing it unplugged in a test routine designed for acoustic guitars, you’d conclude that it’s got a great feel and smooth action, but that it’s way too quiet and sounds unremarkable. You would also be a fool. Same goes for CR’s dead room test of the HomePod.
    edited May 2018 tallest skiltmay
  • Reply 28 of 42
    Mike WuertheleMike Wuerthele Posts: 6,858administrator
    AppleZulu said:
    I was really hoping that the conversation about the HomePod review would have taken CR to task for using a methodology that goes a long way to defeat a key aspect of the device. HomePods analyze the acoustics of the room and modify the signal coming out of the speakers to cancel out undesired effects of the room and they also use acoustically reflective surfaces in the room to actually enhance the sound. CR tests speakers in a room designed to deaden surfaces by absorbing sound waves that would otherwise be reflected. That’s possibly a “fair” way to compare standard speakers to one another, though still not a real-world test. Nobody lives in a place with acoustical tiles all over the room. For the HomePod, that room will defeat precisely the thing that sets the HomePod apart, and that’s not a fair way to assess the device. Compare a HomePod to other speakers in, well, a home, and it will shine where others are diminished.

    If you tested a Rickenbacker 12-string electric guitar by playing it unplugged in a test routine designed for acoustic guitars, you’d conclude that it’s got a great feel and smooth action, but that it’s way too quiet and sounds unremarkable. You would also be a fool. Same goes for CR’s dead room test of the HomePod.
    Here's the thing -- now you know. All the other pictures they've published of the HomePod has been in some nice office setting with artistically applied bokeh.
    hammeroftruth
  • Reply 29 of 42
    hammeroftruthhammeroftruth Posts: 1,303member
    AppleZulu said:
    I was really hoping that the conversation about the HomePod review would have taken CR to task for using a methodology that goes a long way to defeat a key aspect of the device. HomePods analyze the acoustics of the room and modify the signal coming out of the speakers to cancel out undesired effects of the room and they also use acoustically reflective surfaces in the room to actually enhance the sound. CR tests speakers in a room designed to deaden surfaces by absorbing sound waves that would otherwise be reflected. That’s possibly a “fair” way to compare standard speakers to one another, though still not a real-world test. Nobody lives in a place with acoustical tiles all over the room. For the HomePod, that room will defeat precisely the thing that sets the HomePod apart, and that’s not a fair way to assess the device. Compare a HomePod to other speakers in, well, a home, and it will shine where others are diminished.

    If you tested a Rickenbacker 12-string electric guitar by playing it unplugged in a test routine designed for acoustic guitars, you’d conclude that it’s got a great feel and smooth action, but that it’s way too quiet and sounds unremarkable. You would also be a fool. Same goes for CR’s dead room test of the HomePod.
    Here's the thing -- now you know. All the other pictures they've published of the HomePod has been in some nice office setting with artistically applied bokeh.
    So in the end we find out the old adage is true. 
    "Opinions are like @ssholes, everyone has one."

    CR might have started out trying to help the general population avoid products that had inherent design flaws like appliances, but when they branched out to computers during the mid to late 80's, I knew their choices were very questionable. It was downhill from there. 

    The only reason I will occasionally look at them is to see how bad they think something is and compare their review to the rest of the world. 

    I remember when they picked the Neptune washer machine and a couple of years later there was a big class action that forced Maytag to buy back many of those machines. 

    I think it was very gracious of Appleinsider to visit CR and write an article that didn't bash them and explained how they came to their decisions when reviewing products. It confirms to me that they are not the unbiased source that they claim to be. In fact I don't think there is anything that is unbiased anymore, just levels of bias. 

    I think if more people knew that for them, visual aesthetics don't factor in reviews, they would stop using CR as a resource.  The fact that they said that, violates the very notion that they are scientific in their research. Science, rather Biology, proves that aesthetics plays a huge role in our species and planet as a whole. 

  • Reply 30 of 42
    Dave KapDave Kap Posts: 60member
    Thank you for this report. I appreciate the effort you put into it and I’m a huge fan of your site. 

    I wanted to share this article with you because I feel like their methodology is hugely flawed (like I recall Daniel stating in a past article):

    https://www.technightowl.com/2018/05/consumer-reports-product-testing-shortcomings-part-two/
    tmay
  • Reply 31 of 42

    The only "controversy" with regard to CR's evaluation of Apple products is the notion that some Apple enthusiasts have that data doesn't matter, that Apple products are obviously the best products, and that anyone who disagrees with that assessment is "biased against Apple."

    What CR offers is data.  Data isn't the entire story, but it's definitely part of the story, and should be considered.  What one actually does with that data is up to the individual.  I've been reading and relying on CR since the early 70's, and while I certainly don't always agree with their recommendations, I've never seen any reason to deny the overall validity of their statistical data. 

      "Aesthetics" isn't data.  It's almost completely subjective.  And while it's certainly valuable to some, it's less valuable, even unimportant to others.  I for one, don't give a rat's ass what a speaker looks like, as long as it reproduces the sound I want.  I've never once considered the aesthetics of an iPhone when purchasing, I buy iPhones because they bloody well work, and work the way I want them to, i.e. reliably.

  • Reply 32 of 42
    sdw2001sdw2001 Posts: 18,015member
    Consumer Reports is garbage.  It's not just Apple...they've been giving bogus ratings and reviews for years.  It's not bias, it's incompetence and myopia.  They don't understand that there is more to any product than specs.  They openly said they don't consider aesthetics.  What about the way the keyboard feels?  What about how it feels to rest your hands.  The clicking motion?  Ease of gestures with multi-touch?  

    As for incompetence:  So you run one test, and you get 10 hours.  You run another and get 4.  Then you run a third test and get something else.  Instead of wondering if something you're doing is affecting the result, you write the review and refuse to recommend it?  You do this despite Apple saying "this shouldn't be happening" and while they are conducting their own investigation?   That's absurd.  
    edited May 2018 glee217
  • Reply 33 of 42
    EsquireCatsEsquireCats Posts: 1,268member
    "We treat every company and every product the same way. And it's been like that since 1936"
    Consumer Union's responses here are total crap. (Consumer Union is the organisation that runs Consumer Reports.)

    If they noticed something was wrong with the MacBook Pro testing, as they stated, then why rush to publish what is obviously wrong data.
    Claiming to test thousands of devices is not a sufficient credential If their test methodology was to give up testing a device after just three attempts, each of which resulted in significantly lower performance than the previous test. Critically it looks more like their goal was to see how small they could get the battery performance and then publish that data to get a controversial headline, a practice which is not unique to this incident


    So let's use a non-Apple example then. In the youtube video linked we have extracts from court tapes of when Suzuki sued Consumer Union for falsely reporting that the Suzuki Samurai was more prone to rollover accidents than other SUVs - a total fabrication. CR's results could not be reproduced by any other testing agency worldwide. The National Highway Traffic Safety Administration went so far as to reject the conclusions reported by Consumer Reports.
    edited May 2018 hammeroftruth
  • Reply 34 of 42
    chasmchasm Posts: 3,275member
    The purpose of this visit (I thought) was to resolve two controversies:

    1. Why they turned caching off on the MacBook Pro when that’s not how consumers would use the machine. I’m glad they did in a way, it exposed a bug that Apple needed to fix, but why did CR do that in the first place?

    2. Why did they do a “first look“ (poorly tested) article on the HomePod without doing similar first-impression testing on its primary competitors? Doesn’t that foster an impression of bias? Isn’t “first look” type stuff a bad idea entirely in a mag known for rigorous high-quality testing, i.e. their version of “clickbait?”

    Sadly, both of these questions remain unanswered — leaving people so inclined with zero refutation of a curious Apple-focused exception policy by CR. I am glad that CR clarified that they ignore aesthetics in their evaluations, it explains a lot of their reviewing (and odd biases in other areas alongside their Apple bias), and confirms what I’ve long suspected — their approach to the most valuable component of tech (the overall experience) is fundamentally flawed and aimed purely at spec-whores.

    That approach can be useful when all the items you might be evaluating our commodity devices, like car radar detectors — I get that. But it’s fairly useless when you’re comparing devices where the aesthetic is the driving differentiator — or when other non-tech factors, like privacy and after-sale support, are equally important.

    I am grateful that this series has clarified for me that CR is useful for some types of reviews, but useless for others, but I’m disappointed the two straightforward questions I asked were not addressed.
    edited May 2018
  • Reply 35 of 42
    glee217glee217 Posts: 15member
    I've been a CR online subscriber for years. I often don't agree with their assessments, but I think it's important that someone do this sort of independent testing and they are the only option. So, I support them with my subscription. That said, I wouldn't buy or not buy anything based exclusivity on a CR test report. It's a data point. One that I would rate much higher than a manufacturer's marketing blurb or your typical Amazon review, but still just one data point. To me it's worth the minimal cost of the CR subscription.
     Yeah I would use consumer reports along with other reviews from another source I will not just count alone on consumer reports. By the way I think I paid $20 for my online subscription if I  remembered. 
  • Reply 36 of 42
    Mike WuertheleMike Wuerthele Posts: 6,858administrator
    chasm said:
    The purpose of this visit (I thought) was to resolve two controversies:

    1. Why they turned caching off on the MacBook Pro when that’s not how consumers would use the machine. I’m glad they did in a way, it exposed a bug that Apple needed to fix, but why did CR do that in the first place?

    2. Why did they do a “first look“ (poorly tested) article on the HomePod without doing similar first-impression testing on its primary competitors? Doesn’t that foster an impression of bias? Isn’t “first look” type stuff a bad idea entirely in a mag known for rigorous high-quality testing, i.e. their version of “clickbait?”

    Sadly, both of these questions remain unanswered — leaving people so inclined with zero refutation of a curious Apple-focused exception policy by CR. I am glad that CR clarified that they ignore aesthetics in their evaluations, it explains a lot of their reviewing (and odd biases in other areas alongside their Apple bias), and confirms what I’ve long suspected — their approach to the most valuable component of tech (the overall experience) is fundamentally flawed and aimed purely at spec-whores.

    That approach can be useful when all the items you might be evaluating our commodity devices, like car radar detectors — I get that. But it’s fairly useless when you’re comparing devices where the aesthetic is the driving differentiator — or when other non-tech factors, like privacy and after-sale support, are equally important.

    I am grateful that this series has clarified for me that CR is useful for some types of reviews, but useless for others, but I’m disappointed the two straightforward questions I asked were not addressed.
    They got answered, but perhaps not the way you wanted them to be.
  • Reply 37 of 42
    glee217glee217 Posts: 15member
    sdw2001 said:
    Consumer Reports is garbage.  It's not just Apple...they've been giving bogus ratings and reviews for years.  It's not bias, it's incompetence and myopia.  They don't understand that there is more to any product than specs.  They openly said they don't consider aesthetics.  What about the way the keyboard feels?  What about how it feels to rest your hands.  The clicking motion?  Ease of gestures with multi-touch?  

    As for incompetence:  So you run one test, and you get 10 hours.  You run another and get 4.  Then you run a third test and get something else.  Instead of wondering if something you're doing is affecting the result, you write the review and refuse to recommend it?  You do this despite Apple saying "this shouldn't be happening" and while they are conducting their own investigation?   That's absurd.  
    sdw2001 said:
    Consumer Reports is garbage.  It's not just Apple...they've been giving bogus ratings and reviews for years.  It's not bias, it's incompetence and myopia.  They don't understand that there is more to any product than specs.  They openly said they don't consider aesthetics.  What about the way the keyboard feels?  What about how it feels to rest your hands.  The clicking motion?  Ease of gestures with multi-touch?  

    As for incompetence:  So you run one test, and you get 10 hours.  You run another and get 4.  Then you run a third test and get something else.  Instead of wondering if something you're doing is affecting the result, you write the review and refuse to recommend it?  You do this despite Apple saying "this shouldn't be happening" and while they are conducting their own investigation?   That's absurd.  
    Yeah thats why Apple Insiders said CR should speak with Apple on what the issue(s) are so Apple can address the “this shouldn’t be happening” before publishing the ratings/review. 
  • Reply 38 of 42
    joeljrichardsjoeljrichards Posts: 23unconfirmed, member
    But how did they actually test the HomePod? My concern is that CR only tests products based on the capabilities of existing products. The biggest selling point of the HomePod is not how it sounds great when sitting in the sweet spot in a treated listening environment. The HomePod is amazing because it sounds almost as good no matter where you are and no matter where you put it. 

  • Reply 39 of 42
    AppleZuluAppleZulu Posts: 1,989member

    The only "controversy" with regard to CR's evaluation of Apple products is the notion that some Apple enthusiasts have that data doesn't matter, that Apple products are obviously the best products, and that anyone who disagrees with that assessment is "biased against Apple."

    What CR offers is data.  Data isn't the entire story, but it's definitely part of the story, and should be considered.  What one actually does with that data is up to the individual.  I've been reading and relying on CR since the early 70's, and while I certainly don't always agree with their recommendations, I've never seen any reason to deny the overall validity of their statistical data. 

      "Aesthetics" isn't data.  It's almost completely subjective.  And while it's certainly valuable to some, it's less valuable, even unimportant to others.  I for one, don't give a rat's ass what a speaker looks like, as long as it reproduces the sound I want.  I've never once considered the aesthetics of an iPhone when purchasing, I buy iPhones because they bloody well work, and work the way I want them to, i.e. reliably.

    Consumer Reports doesn’t just offer data. Their primary product is in fact summary conclusions and summary product comparisons. 

    My concern, with the HomePod test and review as an example, is the methodology used to gather their data upon which they base their summary conclusions. In the case of the HomePod, it appears CR made incorrect assumptions about the device - that it is a standard mechanical speaker - and performed a testing regimen designed to create an acoustically neutral environment so that they can compare sterile results with other standard mechanical speakers. By doing this, their data is skewed and flawed, and their conclusions misleading. Unlike all the other speakers they compared it to, the HomePod uses a computer and an array of microphones to take audio measurements of the room, and then adjust output to actually incorporate acoustically reflective surfaces to both cancel out unwanted echoes and also build a presence or soundstage. CR’s testing room eliminates the normal real-world environment by covering reflective ceilings and walls with sound-absorbing acoustical tiles or foam. While that may provide a “neutral” testing environment for standard, unidirectional speakers, it specifically defeats the exact thing that Apple spent considerable time and resources to bake into the HomePod, making it fundamentally different than anything a non-audiophile home consumer has ever been able to buy. With a pair of standard speakers, you get a “sweet spot” in front of and centered between them that is better than anywhere else in the room. With a HomePod in a normal environment, the “sweet spot” is virtually anywhere in the room. That’s going to be lost in CR’s testing environment.

    So sure, CR uses data to create their conclusions, but by failing to consider or understand innovative technology, their data is fundamentally flawed. Most consumers aren’t going to understand that, and will only read the summary conclusions, and thus think that the HomePod is unremarkable as a listening device. 

    As an aside, I would go further to argue that the acoustically deadened room, if used as the only test for speakers, is a flawed methodology overall. It does offer a certain neutral comparison that is probably useful, but only as a part of their analysis. It provides data on the raw frequency response of the speaker. Very few home consumers listen to their speakers in such a neutral environment, however. Even ‘dumb’ speakers will perform differently based on placement in a normal room. Speakers placed in the corners of a room, or in wooden bookshelves will sound vastly different than they do isolated on a shelf in a room covered with sound-absorbing acoustical tiles. A speaker that CR rates as sounding small and underwhelming could in fact yield a rich, full sound in a real-world setup, or a speaker CR reports as having a good, flat frequency response could end up sounding boomy and muddy in the corners of a normal listener’s living room.

    Clamoring to be “fair” and “neutral” by intentionally ignoring innovative technology designed to react to and perform in real-world environments ends up not being fair or neutral at all. It instead favors those things that perform best in a sterile environment and don’t take into consideration real world environments. 
  • Reply 40 of 42
    sglewissglewis Posts: 3member
    nunzy said:
    They downplay the stuff that really matters, like industrial design and ease-of-use. Who really cares how good a speaker sounds if it is hard to set up and looks ugly?
    Beauty is in the eye of the beholder. And more importantly, it's easy to discern. I can walk into a Best Buy and see how easy a speaker is to use. I can also judge by my merits how nice it looks. But in that environment, I sure as heck can't judge audio quality.

    Also, what do you mean they "downplay the stuff that really matters"? Matters to who? I buy certain products based on their looks. Other products, I could care less what it looks like, but want to know how it performs. But in any case where looks matter, I can form an opinion easily from a photo on a webpage or a demo product on a shelf. 
Sign In or Register to comment.