Inside Consumer Reports: How iPhone, iPad, Mac, and HomePod testing is performed

2»

Comments

  • Reply 21 of 30
    genovellegenovelle Posts: 1,480member
    blastdoor said:
    nunzy said:
    Consumer Reports hates Apple. They have been completely wrong in every single one of their reviews.
    Pretty sure various Macbooks and Macs have been recommended by Consumer Reports over the years. Are those wrong as well?
    I was about to say the same thing. 

    CR is capable of being wrong, but I think their methods and intentions are basically good. 

    "hate" definitely exists, but it's not coming from CR. 
    I believe the hate stems from which manufacturers the reviewers prefer. Having an open ongoing line of communication with the companies you review unbiased is not realistic because humans develop relationships and perceptions are colored by the influence of people they like. The fact that they mention the like that the manufacturers keep them up to date on what’s coming May explain why they may have a preference for other companies. Like pushing antennagate when holding most smartphones on the market at the time provided similar result in tests. Apple’s secretive nature does not seam to fit their preferred relationship with companies. I noticed a similar bias from them against Honda who also tends to less forthcoming on their product roadmap than competitors. 
  • Reply 22 of 30
    Hmmm. "Fake Reviews!?!"
  • Reply 23 of 30
    MplsPMplsP Posts: 3,925member
    To be honest, there are many people (and quite a few here on AI) for whom anyone that dares utter a negative word about anything Apple is wrong, a Hater, or otherwise some form of idiot that can’t be trusted to evaluate a rock. 


    gatorguymuthuk_vanalingam
  • Reply 24 of 30
    bluefire1bluefire1 Posts: 1,302member
    I've subscribed to CR for decades and they've been in Apple's corner for most of those years. Overall, their ratings are very fair, honest and honorable.
    welshdoggatorguy
  • Reply 25 of 30
    GeorgeBMacGeorgeBMac Posts: 11,421member
    dewme said:
    I have no problem with product evaluations that involve objective measurements and data comparisons that are relevant to the products being evaluated. Data is data and the scientific method should prevail.
    ....
    Your faith may be misplaced...
    Perhaps an analogy is suitable to that blanket statement:
    Yesterday I was reading a study in a reputable medical journal that "proved" that eating eggs is not unhealthy for those with diabetes...   Huh?   That flies in the face of most modern scientific thought.   Then, reading a little deeper into it, it became apparent that they were testing those whose cardiac risk factors were already through the roof.  So increasing any one of them made little or no difference -- like throwing a bucket of water into the ocean.   Then, it all became clear at the end when it was acknowledged that the study was paid for by the "Australian Egg Company" and conducted by researchers who had accepted consultant's fees from the company...

    So much for scientific, objective research.
    Medical, healthcare studies are some of the reputable and scientific in the world.  Yet, most medical professionals are increasingly aware that even those supposedly impartial, objective studies conducted with impeccable procedures can prove pretty much anything the funders of that study want it to prove...

    And another:  My first career was as an accountant.  My eyes were opened when an executive asked my boss in April whether the plant would show a profit during its annual June shutdown.  His answer:  "We decided to show a loss of $100K."   It's nice when you can cook the books to show whatever you decide...   (I left the company shortly after...)

    The lesson is:  "Figures (and Data) lie, and Liars figure"

    I place honest and impartial over any method -- including the so called scientific one.
    muthuk_vanalingam
  • Reply 26 of 30
    jcs2305jcs2305 Posts: 1,337member
    mdwychoff said:
    Hmmm. "Fake Reviews!?!"
    Noooooooooo! 
  • Reply 27 of 30
    dewmedewme Posts: 5,363member
    dewme said:
    I have no problem with product evaluations that involve objective measurements and data comparisons that are relevant to the products being evaluated. Data is data and the scientific method should prevail.
    ....
    Your faith may be misplaced...
    Perhaps an analogy is suitable to that blanket statement:
    Yesterday I was reading a study in a reputable medical journal that "proved" that eating eggs is not unhealthy for those with diabetes...   Huh?   That flies in the face of most modern scientific thought.   Then, reading a little deeper into it, it became apparent that they were testing those whose cardiac risk factors were already through the roof.  So increasing any one of them made little or no difference -- like throwing a bucket of water into the ocean.   Then, it all became clear at the end when it was acknowledged that the study was paid for by the "Australian Egg Company" and conducted by researchers who had accepted consultant's fees from the company...

    So much for scientific, objective research.
    Medical, healthcare studies are some of the reputable and scientific in the world.  Yet, most medical professionals are increasingly aware that even those supposedly impartial, objective studies conducted with impeccable procedures can prove pretty much anything the funders of that study want it to prove...

    And another:  My first career was as an accountant.  My eyes were opened when an executive asked my boss in April whether the plant would show a profit during its annual June shutdown.  His answer:  "We decided to show a loss of $100K."   It's nice when you can cook the books to show whatever you decide...   (I left the company shortly after...)

    The lesson is:  "Figures (and Data) lie, and Liars figure"

    I place honest and impartial over any method -- including the so called scientific one.
    I think we are in total agreement when you expand on the underlying reasoning. One of the definitions of "objective" according to Merrium-Webster:

    expressing or dealing with facts or conditions as perceived without distortion by personal feelings, prejudices, or interpretations  objective artan objective history of the warobjective judgment
    b of a test limited to choices of fixed alternatives and reducing subjective factors to a minimum  Each question on the objective test requires the selection of the correct answer from among several choices.

    A study sponsored by an egg company to highlight the supposed health benefits of eating eggs is very likely to be subjected to distortion, bias, and creative presentation. But I do understand your concern that numbers, statistics, and results alone can never be taken at face value without knowing the details behind the number. More so if they can not be trusted to be generated from objective reasoning and a repeatable methodology.

    My point was along the lines of separating objective measures (with all the qualifiers) like height, weight, speed, volts, decibels, etc., from subjective interpretations like "full bodied sound," "pleasing to the ear," "feels snappy and responsive," etc., types of evaluations. 

    To add to your point about various forms of deception, those of us who believe in science, engineering, and well behaved systems always find a level of comfort in knowing that these things almost always fully resolve themselves to math, physics, thermodynamic, etc., principles that can be independently verified & validated. People who are at the pinnacle of trust and implied competency in these fields have decades of education and training in hard science, mathematics, and engineering to prepare themselves to work in these fields. But we often overlook the fact that all this hard science and reasoning is perched atop a slushy mess of human psychology with all of its inherent weaknesses and cognitive biases. How many hard scientists are as grounded in psychology as they are in physics or mathematics?

    At the very least we're dealing with a 2-variable truth table: science and scientist. So we have four possible combinations of Good Science, Bad Science, Good Scientist, and Bad Scientist. Only one of these four possible combinations fits the definition of "objectivity."
    GeorgeBMac
  • Reply 28 of 30
    GeorgeBMacGeorgeBMac Posts: 11,421member
    dewme said:
    dewme said:
    I have no problem with product evaluations that involve objective measurements and data comparisons that are relevant to the products being evaluated. Data is data and the scientific method should prevail.
    ....
    Your faith may be misplaced...
    Perhaps an analogy is suitable to that blanket statement:
    Yesterday I was reading a study in a reputable medical journal that "proved" that eating eggs is not unhealthy for those with diabetes...   Huh?   That flies in the face of most modern scientific thought.   Then, reading a little deeper into it, it became apparent that they were testing those whose cardiac risk factors were already through the roof.  So increasing any one of them made little or no difference -- like throwing a bucket of water into the ocean.   Then, it all became clear at the end when it was acknowledged that the study was paid for by the "Australian Egg Company" and conducted by researchers who had accepted consultant's fees from the company...

    So much for scientific, objective research.
    Medical, healthcare studies are some of the reputable and scientific in the world.  Yet, most medical professionals are increasingly aware that even those supposedly impartial, objective studies conducted with impeccable procedures can prove pretty much anything the funders of that study want it to prove...

    And another:  My first career was as an accountant.  My eyes were opened when an executive asked my boss in April whether the plant would show a profit during its annual June shutdown.  His answer:  "We decided to show a loss of $100K."   It's nice when you can cook the books to show whatever you decide...   (I left the company shortly after...)

    The lesson is:  "Figures (and Data) lie, and Liars figure"

    I place honest and impartial over any method -- including the so called scientific one.
    I think we are in total agreement when you expand on the underlying reasoning. One of the definitions of "objective" according to Merrium-Webster:

    expressing or dealing with facts or conditions as perceived without distortion by personal feelings, prejudices, or interpretations  objective artan objective history of the warobjective judgment
    b of a test limited to choices of fixed alternatives and reducing subjective factors to a minimum  Each question on the objective test requires the selection of the correct answer from among several choices.

    A study sponsored by an egg company to highlight the supposed health benefits of eating eggs is very likely to be subjected to distortion, bias, and creative presentation. But I do understand your concern that numbers, statistics, and results alone can never be taken at face value without knowing the details behind the number. More so if they can not be trusted to be generated from objective reasoning and a repeatable methodology.

    My point was along the lines of separating objective measures (with all the qualifiers) like height, weight, speed, volts, decibels, etc., from subjective interpretations like "full bodied sound," "pleasing to the ear," "feels snappy and responsive," etc., types of evaluations. 

    To add to your point about various forms of deception, those of us who believe in science, engineering, and well behaved systems always find a level of comfort in knowing that these things almost always fully resolve themselves to math, physics, thermodynamic, etc., principles that can be independently verified & validated. People who are at the pinnacle of trust and implied competency in these fields have decades of education and training in hard science, mathematics, and engineering to prepare themselves to work in these fields. But we often overlook the fact that all this hard science and reasoning is perched atop a slushy mess of human psychology with all of its inherent weaknesses and cognitive biases. How many hard scientists are as grounded in psychology as they are in physics or mathematics?

    At the very least we're dealing with a 2-variable truth table: science and scientist. So we have four possible combinations of Good Science, Bad Science, Good Scientist, and Bad Scientist. Only one of these four possible combinations fits the definition of "objectivity."
    All quite true!
  • Reply 29 of 30
    robjn said:
    They report that they test everything with the same Jazz CD and this certainly adds consistency across tests but when the human brain is hearing music it knows well it can trick the listener by automatically filling in the gaps and making the person ‘hear’ what they expect to hear even though the speaker might not be reproducing certain elements well.

    The scientific test equipment we have picks up problems the human ear cannot easily discern.

    Anyone who was involved in 1990s hi-fi remembers this device fondly:

    https://www.gearsource.com/catalog/stockitem/audiocontrol-sa3051-real-time-spectrum-analyzer
    It's simple, sure but it was state-of-the-art (especially for the ~$900 it was priced at) for measuring the effect of positional, component or environmental change on generated pink noise, and AudioControl is still selling a similar item (the mildly updated SA-3052) today.
    edited May 2018
  • Reply 30 of 30
    Just for the record, Mark Connelly, the organization's overall director of testing, doesn't have a "P.E. degree," as there is no such animal.  A P.E. is a Professional Engineer, licensed by the state after rigorous testing.  Mr. Connelly almost certainly also has a degree in some flavor of engineering; that is usually required, along with experience, before someone can sit for the test.
Sign In or Register to comment.