Consumer Reports' dismissal of HomePod a familiar tale to Apple fans [u]

1246

Comments

  • Reply 61 of 109
    I gave up on Consumer Reports years ago.  Knew they were biased against Apple from the start.  But why I quit them originally was on their biased reporting in favor of Sears products back in the day.  Especially Kenmore.  Lost several hundred dollars by following their Recommendations.
    albegarcpscooter63
  • Reply 62 of 109
    foggyhillfoggyhill Posts: 4,767member
    wizard69 said:
    Boy a lot of butt hurt people here.   

    I havent even seen a HomePid yet because frankly it doesnt even come close to meeting my requirements.   As such i cant comment on sound quality.  What i can say though is that everybodies hearing is a bit different so there are no absolutes as to which is better.  

    As for some of the sites refrenced that supposedly reviewed the HomePod why are people yere so willing to trust those positive reviews?    Seriously guys why throw your opinion over to the side that gets stuff to review with the expectation that the review will be positive?     Im not here to say that CRs reports and testing is all that can be,   Rather id trust them far more than engadget and a bunch of other sites with their glowing reviews.  

    In the end it is how the speaker sounds in the installation it is used in that counts.   That is dependent on many factors that makes much of the testing and reviews worthless in my opinion.    A common example here is testing frequency response of a speaker and declaring it good or bad baseed on the results.  While a flat response is nice technically HomePod and many other speaker installations have their response patterns altered via equalization so no flat response.  HomePod is apparently worse here as it is doing some equalization dynamically.   

    Im not trying to dismiss speaker specs here just trying to point out that how a speaker ultimately sounds to a person is subjective.   More so that sound is very installation dependent.  As such dont get wrapped up in metrics as that is only part of the story.  
    You should at stopped at the fracking "I haven't seen a Homepod", everything else is pure logorea.
    tmayargonaut
  • Reply 63 of 109
    tonkinitetonkinite Posts: 7unconfirmed, member
    CR seemed good a decade or more ago. As technology has progressed, they seem like they’ve lost their way. The testing they do on anything that has circuitry seems poorly thought out, incomplete. They have definitely developed outright biases in some areas other than tech,, like medicine & food. It’s almost like they’ve become Luddites.
  • Reply 64 of 109
    maestro64 said:
    Okay I find this totally funny, My family also have a Gremlin and my sister and I beat the hell out of that car, patch it together with bondo due to all the rust and had a front end accident and just pulled the bumper out and that things never died, it just kept running. 
    I find it funny that anybody, much less CR, thought a car named Gremlin would be worry-free. It's like Lucas electrics (Prince of Darkness) light bulbs...the name tells you what to expect. 

    But it was the best that Kenosha could do at the time. When I used to drive past the plant there would be a sea of Gremlins awaiting gullible dealers to place an order.


    argonautrandominternetperson
  • Reply 65 of 109
    dewme said:
    I understand that Consumer Reports' entire business model used to be predicated being a relevant, trustworthy, and objective evaluator of consumer products. That was a very lofty and admirable set of goals but they have not always been up to the task and have skewed off the path of objectivity many a time. 
    Back in the day, 1950s and 60s, Consumer Reports used to hire actual scientists and engineers, and they got paid pretty well. My older brother the physicist worked for them for several months while waiting until his Bell Labs position opened up. CR used to do rigorous testing and analysis. Now all that is an echo. Their car reliability survey is still the best in the auto world (and by orders of magnitude the largest), but most other things are not subject to the same level of analysis.
    randominternetperson
  • Reply 66 of 109
    CR is about is credible on Apple products as ZDNet, John C. Dvorak or Rob Enderle. Which is to say, not credible at all.
  • Reply 67 of 109
    DAalsethDAalseth Posts: 2,783member
    I found their reviews for cars to be utterly useless decades ago when I was buying my first car and haven’t bothered with them ever since.
    Same here. They got on a Japan Is Best thing back in the '80s and '90s. Even when Detroit produced some good cars, CR would trash them. I just ignore CR. There are far better resources online.
  • Reply 68 of 109
    gatorguy said:
    gatorguy said:
    "Apple says that every time you move the speaker, it senses the motion, then automatically adjusts itself to its placement in the room using a series of test tones and complex algorithms to minimize reflections from nearby walls or other objects. That’s not a feature we evaluated."

    ...is such a core aspect of the design...?  Could that explain why they found the sound (or EQ) less to their liking...?
    Is that quote from their actual review?

    Not evaluating a core feature/aspect of a product's design that has a direct bearing on performance is very un-scientific.
    It is an actual quote, yes.
    Mike, since it's something that's done seamlessly it's not something they should have to consider is it? The HomePod should sound its best no matter whether it was moved from one spot to another. 
    I don't have an issue with them not testing that particular feature, and I didn't mention it in the story because it wasn't relevant to the larger point and would inflame for no real reason.

    My issue is, a drive-by assessment by CR with no discussion behind the opinion to try and capture the headline cycle. Why didn't they finish the evaluation, and publish after their customary month?
    With that I agree. I've not read the original CR piece. Did they say they were doing a more complete review this next month? If not it does lean towards a drive-by. 
    "Full test results for these speakers, which also incorporate factors such as ease of use and versatility, will be released in the next few weeks."

    As far as I can tell, this is the first time they've pre-announced results from a test that won't be complete for a few weeks.
    That really does fly in the face of what Consumer Reports claims to do. Publishing a conclusion while offering no reasons for it is of little use to the reader. 

    I’ll make some predictive notes, though, based on what you can see in the little video clip that appears to show their testing room. First, that room is a horrible setup for any speaker. Every set of speakers they tested is piled together on a couple of rows of shelving, and they’re all in a room with sound baffling on the walls and ceiling.

    Second, that’s not a particularly conducive environment for any speaker, but (much like they did in their recent MBP test) they’ve actually created an environment that defeats the HomePod’s key feature. All the sound baffling will kill much of the testing and calibration that the HomePod is programmed to do. Plus, most people’s homes have hard surface walls and ceilings, which the HomePod uses to its advantage. 

    So their testing methodology looks like they compare the speakers’ raw output in a dead space, with the listener positioned right in front of the speaker. Plus the testers have manually adjusted EQ and whatever for the other speakers to try to match them to the output of their reference speakers (wouldn’t this introduce a certain sort of bias?), while the HomePod must simply try to run its programs to create a flat frequency response in a dead room that is unlike most real world environments and is probably a standard deviation or two away from what it’s normally programmed to do. They think they’re setting up a dispassionate laboratory test, but by ignoring the fact that a new device functions in a fundamentally different way, their test is anything but dispassionate and scientific. It entirely misses the point of innovation. 
    edited February 2018 roundaboutnowtmayfoggyhillpscooter63randominternetperson
  • Reply 69 of 109
    CR is like every other review site.  It’s more about confirming what you’ve heard somewhere else.

    For cars I’ll look at CR reliability/safety studies, warranty comparisons etc.  but these are just data points.  For subjective reviews I’ll look at customer experiences, and car expert reviews.  All that just determines which cars to test drive.  Because in the end only my review matters.

    The same goes for audio devices...

    Personally, I have some Panasonic wireless headphones that work fine so spending on a HomePad that doesn’t even connect to the TV is a no go.


    CR is actually useless for car ratings because of confirmation bias.

    For example, they once gave a lower rating for the Pontiac Vibe (reliability) than the Toyota Matrix. They're the same car and made by Toyota. The Pontiac has some trim pieces to make it look different, but all the underpinnings are pure Toyota.

    The reason for this is they rely a lot on data sent to them by owners. Which is useless because people have not only terrible memory but selective memory. If a person who likes Toyota buys one for reliability, and then has to go get something repaired under warranty, their attitude is "Well, looks like I got the one-in-a-million - Toyota fixed it so all's well". If a person buys a brand that's not as reliable, and has to get the same type of issue fixed, their attitude is "Well, I see my car is living up to its reputation as being less reliable." Two owners with the same problem and two completely different opinions about their cars reliability.

    There are even people who claimed the Pontiac Vibe used much more gas than the Toyota Matrix (which is actually impossible). Yet this fact shows up in lots of aggregated reviews from owners of these vehicles.
    roundaboutnowfirelockpscooter63loopless
  • Reply 70 of 109
    dougddougd Posts: 292member
    I don't really care what they say, I wouldn't buy this device. It's too expensive and too Apple centric
  • Reply 71 of 109
    ivanhivanh Posts: 597member
    “The testing was dramatic, with the phones taking major screen damage in a steel tumbling device after 50 rotations. We're not sure how "real world" this is...” 

    In real world, an iPhone may drop 100 times from the top of a long stairs to the bottom of it.
  • Reply 72 of 109

    If you want to get a good understanding of HOW Consumer Reports came out with such a negative assessment of the HomePod, in contradiction to the almost unanimous superlative reviews and testing results done by audio engineers and technology professionals, read this excerpt from CR's "review":

    "Consumer Reports evaluates sound quality for speakers, smart or otherwise, in a dedicated listening room in which our experienced testers compare each model with high-quality reference speakers. Each test unit that allows for user controls is tuned for optimum sound quality—we want the speakers to sound their best."

    Also, take a look at the photo of CR's testing "rig”. It has all speakers crammed together on a multi-shelf stand with many speakers in front of, behind, above, and below each other. It is a very cumbersome setup, in a very unnatural test environment (unlike ANY room and speaker placement that a real person would ever experience).

    In the photo of the Consumer Reports speaker setup in the listening room, notice that the HomePod speaker has no space or flat wall surfaces on the left and right sides, and on the back. The HomePod uses that free space around it, and the rear and side walls to reflect the left and right channels, using audio beamforming, while the center channel (audio that is centered in the sound space) is beamed forward.

    The HomePod is unlike other speakers that just have their internal speakers facing forward, and that DO NOT rely on reflected audio beamforming.

    In Consumer Report's setup, instead of clear, reflected left and right audio channels, the sound would have been muffled. It is unknown whether this debilitating positioning of the HomePod was done purposely, or if it was done due to a lack of understanding by the CR testers.

    In other words, it was a very poorly setup test environment, and the review consisted of "listeners" giving their own subjective feelings about which speaker they liked best. Aside from any unstated motives or biases of the "listeners" at Consumer Reports, there was no methodical or scientific testing done of the speakers, in various "natural" room environments. In contrast, this type of analytical and realistic testing WAS done by other reputable audio engineers and technology professionals.

    DAalsethpscooter63randominternetperson
  • Reply 73 of 109
    I will not comment on CR either way; however, even though I have been a long time Apple fanboy, I have to agree with CRs assessment about the sound quality of the HomePod.  As always, I pre-oreder the new Apple product and was excited to pick it up on Friday.  I got home and set it up and tried to convince myself that it sounded as good as all the reviews said it sounded.  After a couple of days I figured that, perhaps, I got a bum speaker.  I went out a bought a second HomePod and it sounded exactly the same.

    I totally agree with: "the midrange tones were somewhat hazy, meaning that some of the nuance in vocals, guitars, and horns was lost ... Treble sounds, like cymbals, were underemphasized.".  I was also some disappointed with  how loud they would go.

    I'm clearly in the minority.  As a long-time Apple fan, it's disappointing.  I will live with the HomePod a few more days; however, I'm pretty sure I will be returning them this weekend.

    gatorguy
  • Reply 74 of 109
    Mike WuertheleMike Wuerthele Posts: 6,858administrator
    ivanh said:
    “The testing was dramatic, with the phones taking major screen damage in a steel tumbling device after 50 rotations. We're not sure how "real world" this is...” 

    In real world, an iPhone may drop 100 times from the top of a long stairs to the bottom of it.
    Yeah, and in the real world, it could end up in a blender. That doesn't mean that it's realistic -- 100 times from the top of a long stairs to the bottom is a utterly ridiculous assertion that even .01 percent of phones will never see.
    edited February 2018
  • Reply 75 of 109
    David Pogue rated HomePod sound quality very high. Then he did a blind test with random people and HomePod came in last. First was Google Home Max. Just shows how subjective sound is.  
    gatorguy
  • Reply 76 of 109
    I think the problem here is that there is a belief that Consumer Reports, by title alone, sets out to deliver a forecast of the real world expectations of device performance. Yet their analysis is not built from the reports of consumers - far from it.

    "Consumer experience" is the aggregation of the subjective opinions of consumers, the result of this accumulation is an objective fact.

    Instead Consumer Reports continuously engineers unusual test scenarios and apparatuses that do not reflect real world usage; and because of those unusual methods their results are frequently at odds with the average consumer experience. It's also noteworthy that Consumer Reports rushes out sensational stories without conferring with device manufacturers, even when the reported shortcomings are glaring. Headlines are more important than details, and for this Consumer Reports are irresponsible and abusing their position.

    Now we come to a speaker, and Consumer Reports are significantly outmatched here, because a great many people do know how to evaluate speakers performance and such evaluations have come to a starkly different conclusion with the HomePod, both subjectively and objectively with frequency response data along with listening tests.

    So what does this tell us? That it doesn't matter if Consumer Reports favours a product, you should never rely on them for any evaluation because they don't know how to effectively evaluate products.
    pscooter63randominternetperson
  • Reply 77 of 109
    gatorguygatorguy Posts: 24,176member
    David Pogue rated HomePod sound quality very high. Then he did a blind test with random people and HomePod came in last. First was Google Home Max. Just shows how subjective sound is.  
    Just read that myself. Completely blind test and from a not-an-Apple-hater... Pogue says "To be honest I was floored."
    https://finance.yahoo.com/news/head-head-apple-homepod-really-sound-best-160346138.html

    BTW I think Sonos was first and Home Max second, altho the classically trained musician was solid in the Max camp. 
    edited February 2018
  • Reply 78 of 109
    fredtwd said:
    I will not comment on CR either way; however, even though I have been a long time Apple fanboy, I have to agree with CRs assessment about the sound quality of the HomePod.  As always, I pre-oreder the new Apple product and was excited to pick it up on Friday.  I got home and set it up and tried to convince myself that it sounded as good as all the reviews said it sounded.  After a couple of days I figured that, perhaps, I got a bum speaker.  I went out a bought a second HomePod and it sounded exactly the same.

    I totally agree with: "the midrange tones were somewhat hazy, meaning that some of the nuance in vocals, guitars, and horns was lost ... Treble sounds, like cymbals, were underemphasized.".  I was also some disappointed with  how loud they would go.

    I'm clearly in the minority.  As a long-time Apple fan, it's disappointing.  I will live with the HomePod a few more days; however, I'm pretty sure I will be returning them this weekend.

    ...and I’m an android Fanboy.  I owned google smartphones, Samsung watches, google max, etc. for about 17 years now. I totally agree that the HomePod sound quality is clearly superior to all the other speaker systems.  As a long time Android fan, it’s disappointing.  I will live with my Android ecosystem a few more days, however, I’ll be switching entirely to the Apple ecosystem tomorrow including my boycot of Amazon and holding off on a Tesla until Apple car comes to market.  
    randominternetperson
  • Reply 79 of 109
    ivanh said:
    “The testing was dramatic, with the phones taking major screen damage in a steel tumbling device after 50 rotations. We're not sure how "real world" this is...” 

    In real world, an iPhone may drop 100 times from the top of a long stairs to the bottom of it.
    Yeah, and in the real world, it could end up in a blender. That doesn't mean that it's realistic -- 100 times from the top of a long stairs to the bottom is a utterly ridiculous assertion that even .01 percent of phones will never see.
    Right, “may” being the operative word. @ivanh seems to thrive on made up scenarios being disastrous in the “real world”. See his dumb post in the comments of the article about how to turn on AirPlay via Control Center for a point of comparison. 

    Perhaps he works for Consumer Reports as this seems to be what they’re basing their testing on these days: pretend situations.  ;)
  • Reply 80 of 109
    Not on my household, they are like CNN to me, dont care about thier opinions....hehehe. 
    As a consumer, better evaluate it my self🤣
Sign In or Register to comment.