Street test measures Siri comprehension at 83%, accuracy at 62%

124678

Comments

  • Reply 61 of 143

    duplicate
     0Likes 0Dislikes 0Informatives
  • Reply 62 of 143
    gatorguygatorguy Posts: 24,772member

    Quote:

    Originally Posted by jragosta View Post





    Are you a flipping idiot or what?

    I have a PhD in science. I know what is scientific.

    Now, a scientific study may or may not have validity, but if it meets the premises of science, it is a scientific study. That is, it must have:

    1. A testable hypothesis. Check

    2. A methodology for testing the hypothesis. Check.

    3. A control. Check.

    4. Results. Check.

    5. A conclusion. Check.

    It meets the criteria. They are being careful to ensure that it's not being used as the be-all and end-all of research on the subject, but to say that it's not scientific just because it doesn't have a lot of replication is just plain wrong.

    And, no matter how you slice it, it's infinitely more valid than a Motorola advertisement saying that Motorola's phones are better than Apple's phones. That should be obvious to anyone thinking at above a 5 year old's level. Sadly, that seems to leave you out.


    Conclusion? The authors said there isn't a scientifically valid one. Controls?? What controls?? Were the readers aware of what phones were being used and thus intentionally/inadvertantly read to one device differently or at a higher/lower volume. Were all recordings at the same time or tested individually under varying conditions? Were they all new phones with properly working microphones? Was the noise level kept at a  constant level or more the traditional office rise and fall of voices, machinery and phones? Do you know the answers to any of that?


     


    Sir, if you really have a PhD in science (which area BTW) you should use it as a basis for questioning rather than imagining the study you're so entralled with, likely done by a couple of guys with little/no scientific background in some day-trading office, was a "scientific study". You're doing your school a disservice.


     


    Here's the "study" authors. They're stock traders for cryin' out loud, perhaps with vested interests in Apple for all you know. Scientists question. You sir are not a scientist, at least here, no matter what your piece of paper might or might not say.


    http://thearorareport.com/

     0Likes 0Dislikes 0Informatives
  • Reply 63 of 143
    ankleskaterankleskater Posts: 1,287member

    Quote:

    Originally Posted by caliminius View Post


    Why is it inane? Other than you obviously not liking the result?


     


    At the end of the day, they're both search engines. Siri translates the voice input to text and then submits it to a search engine. The accuracy of those searches was vastly in favor of Google. Deal with it.


     


    Siri is a fun feature but its overall performance is lacking. It's great at the simple tasks, like setting alarms, providing directions home, voice dialing, but when you tell it to search for things it really does nose dive into uselessness.





    Such a comparison does not make sense. There's no way it would have been published in a peer-reviewed journal. A proper test would compare apples to apples, with control built in. If it makes sense to compare accuracy of speech processing to text input, why not include sign language and smoke signals too?


     


    This has nothing to do with the utility of Siri. The reported test just does not pass any meaningful scientific scrutiny.

     0Likes 0Dislikes 0Informatives
  • Reply 64 of 143
    ankleskaterankleskater Posts: 1,287member

    Quote:

    Originally Posted by Gatorguy View Post


    Controls?? What controls?? Were the readers aware of what phones were being used and thus intentionally/inadvertantly read to one device differently or at a higher/lower volume. Were all recordings at the same time or tested individually under varying conditions? Were they all new phones with properly working microphones? Was the noise level kept at a  constant level or more the traditional office rise and fall of voices, machinery and phones? Do you know the answers to any of that?


     


    Sir, if you have a PhD in science (which area BTW) you should use it as a basis for questioning rather than imagining the study you so entralled with, likely done by a couple of guys with little/no scientific background in some day-trading offic,e was a "scientific study". You're doing your school a disservice.


     


    Here's the "study" authors. They're stock traders!


    http://thearorareport.com/





    Indeed, their profession makes them ill-qualified to conduct proper testing. But, in this case, the comparison does not even pass the test of common sense.

     0Likes 0Dislikes 0Informatives
  • Reply 65 of 143
    johndoe98johndoe98 Posts: 278member

    Quote:

    Originally Posted by jragosta View Post





    Are you a flipping idiot or what?

    I have a PhD in science. I know what is scientific.

    Now, a scientific study may or may not have validity, but if it meets the premises of science, it is a scientific study. That is, it must have:

    1. A testable hypothesis. Check

    2. A methodology for testing the hypothesis. Check.

    3. A control. Check.

    4. Results. Check.

    5. A conclusion. Check.

    It meets the criteria. They are being careful to ensure that it's not being used as the be-all and end-all of research on the subject, but to say that it's not scientific just because it doesn't have a lot of replication is just plain wrong.

    And, no matter how you slice it, it's infinitely more valid than a Motorola advertisement saying that Motorola's phones are better than Apple's phones. That should be obvious to anyone thinking at above a 5 year old's level. Sadly, that seems to leave you out.


     


    What was the hypothesis? That one phone will perform better? What predictions were involved, and what new phenomena is now capable of being explained by the hypothesis that was not capable of being explained previously, hence requiring this hypothesis?

     0Likes 0Dislikes 0Informatives
  • Reply 66 of 143
    ankleskaterankleskater Posts: 1,287member

    Quote:

    Originally Posted by kent909 View Post


    I think this is pretty impressive. My wife does not comprehend me and she is rarely right.  





    But, unlike your wife, you would dare to tell Siri that she was wrong. :)

     0Likes 0Dislikes 0Informatives
  • Reply 67 of 143
    ankleskaterankleskater Posts: 1,287member

    Quote:

    Originally Posted by Gatorguy View Post


    The articles' authors (who are investment traders, possibly vested in Apple for all you know) say the study "was not scientific" and "no... conclusions should be drawn".



     


    These are perfunctory (even legal?) liability-shedding statements. The fact is that they publish these reports aimed to planting conclusions in people's heads. Otherwise, what purpose does the report serve? 

     0Likes 0Dislikes 0Informatives
  • Reply 68 of 143
    mstonemstone Posts: 11,510member

    Quote:

    Originally Posted by jragosta View Post





    Are you a flipping idiot or what?

    I have a PhD in science. I know what is scientific.

    Now, a scientific study may or may not have validity, but if it meets the premises of science, it is a scientific study. That is, it must have:

    1. A testable hypothesis. Check

    2. A methodology for testing the hypothesis. Check.

    3. A control. Check.

    4. Results. Check.

    5. A conclusion. Check.

    It meets the criteria. 



    ...thinking at above a 5 year old's level. Sadly, that seems to leave you out.


     


    1. Arrogant. Check


    2. Rude. Check


    3. Insulting. Check


    4. Obnoxious. Check


    5. Ass hat. Check

     0Likes 0Dislikes 0Informatives
  • Reply 69 of 143
    gatorguygatorguy Posts: 24,772member

    Quote:

    Originally Posted by ankleskater View Post


     


    These are perfunctory (even legal?) liability-shedding statements. The fact is that they publish these reports aimed to planting conclusions in people's heads. Otherwise, what purpose does the report serve? 



    I completely agree. They had some purpose in mind by publishing it, and it wasn't in the interest of science IMHO.

     0Likes 0Dislikes 0Informatives
  • Reply 70 of 143
    apple ][apple ][ Posts: 9,233member


    SIRI and Apple's dictation is not 100% perfect of course, but it works pretty good, and it's obviously only going to get better as Apple keeps on improving it.


     


    It seems to me that most of the people who have problems with SIRI are people with speech defects, foreigners with ridiculous accents and illegal aliens. Have you ever seen some of the youtube videos of certain people trying to use SIRI? SIRI is not the problem, it's those people and their laughable voices along with their incomprehensible and embarrassing pronunciations.

     0Likes 0Dislikes 0Informatives
  • Reply 71 of 143
    jragostajragosta Posts: 10,473member
    mstone wrote: »
    I've been using Google's voice recognition for quite a while and it works pretty well in both English and Spanish. I just started using Apple's and it seems to be on par with English but so far Spanish is not supported. I'm looking forward to the release later this year.

    That's an entirely different issue.

    It should recognize properly spoken English better than poorly spoken English. Similarly, it would recognize properly spoken Spanish better than poorly spoken Spanish. It's ability to recognize multiple languages doesn't mean that it will be capable of recognizing words that are garbled and mis-spoken in any given language.
     0Likes 0Dislikes 0Informatives
  • Reply 72 of 143
    anonymouseanonymouse Posts: 7,123member

    Quote:

    Originally Posted by Gatorguy View Post


    The articles' authors (who are investment traders, possibly vested in Apple for all you know) say the study "was not scientific" and "no... conclusions should be drawn". Jragosta says the study was scientific and comparisons can easily be made. Who do you think has more validity, the guys that wrote the article or you?  I vote neither, tho the traders haven't yet proved to me that they're dishonest.  


     


    Geesh, yet another instance where Jr can't be wrong. Yet here you are...



     


    Although, regardless of how "scientific" this study was,  it certainly carries more validity than GG's Motorola ad. Seriously, GG, that's an all new low even for you to try to use an ad as proof of such a claim.


     


    However, although I haven't looked at jragosta's cited study, disclaimers by the authors that it wasn't scientific, or that no conclusions should be drawn don't necessarily invalidate it. It's possible that their study was methodologically correct with significant results, but either a) they didn't realize it was or b) as investment traders they made disclaimers as a matter of course so no one could sue them because investment decisions based on the results didn't turn out well. Even professional scientists will sometime caution against basing action on the results of seemingly valid studies.

     0Likes 0Dislikes 0Informatives
  • Reply 73 of 143
    macbook promacbook pro Posts: 1,605member
    "Apple wrote:
    [" url="/t/151000/street-test-measures-siri-comprehension-at-83-accuracy-at-62/40#post_2137151"]SIRI and Apple's dictation is not 100% perfect of course, but it works pretty good, and it's obviously only going to get better as Apple keeps on improving it.

    It seems to me that most of the people who have problems with SIRI are people with speech defects, foreigners with ridiculous accents and illegal aliens. Have you ever seen some of the youtube videos of certain people trying to use SIRI? SIRI is not the problem, it's those people and their laughable voices along with their incomprehensible and embarrassing pronunciations.

    My wife is born and raised Japanese, listening to her trying to coax information from Siri in English is hysterically funny. Siri in Japanese seems to be fairly reliable for her though.
     0Likes 0Dislikes 0Informatives
  • Reply 74 of 143
    apple ][apple ][ Posts: 9,233member

    Quote:

    Originally Posted by MacBook Pro View Post





    My wife is born and raised Japanese, listening to her trying to coax information from Siri in English is hysterically funny. Siri in Japanese seems to be fairly reliable for her though.


    We have something in common, because my gf is also Japanese. :)

     0Likes 0Dislikes 0Informatives
  • Reply 75 of 143
    ankleskaterankleskater Posts: 1,287member

    Quote:

    Originally Posted by Gatorguy View Post


    I completely agree. They had some purpose in mind by publishing it, and it wasn't in the interest of science IMHO.





    Definitely not for "scientific" purposes but rather for promotional purposes. After that, these folks promote themselves as "researcher analysts". With that job title, they do need to provide rigorous analysis, just as pollsters do.

     0Likes 0Dislikes 0Informatives
  • Reply 76 of 143
    irelandireland Posts: 17,802member


    you are one person

     0Likes 0Dislikes 0Informatives
  • Reply 77 of 143
    mstonemstone Posts: 11,510member

    Quote:

    Originally Posted by Apple ][ View Post


    It seems to me that most of the people who have problems with SIRI are people with speech defects, foreigners with ridiculous accents and illegal aliens. Have you ever seen some of the youtube videos of certain people trying to use SIRI? SIRI is not the problem, it's those people and their laughable voices along with their incomprehensible and embarrassing pronunciations.



     


    Quote:

    Originally Posted by Apple ][ View Post


    We have something in common, because my gf is also Japanese. :)



    Does she know how you feel about foreigners?

     0Likes 0Dislikes 0Informatives
  • Reply 78 of 143
    mstonemstone Posts: 11,510member

    Quote:

    Originally Posted by jragosta View Post





    That's an entirely different issue.

    It should recognize properly spoken English better than poorly spoken English. Similarly, it would recognize properly spoken Spanish better than poorly spoken Spanish. It's ability to recognize multiple languages doesn't mean that it will be capable of recognizing words that are garbled and mis-spoken in any given language.


    What issue did you read into my post? There is no issue.

     0Likes 0Dislikes 0Informatives
  • Reply 79 of 143
    apple ][apple ][ Posts: 9,233member

    Quote:

    Originally Posted by mstone View Post


     


    Does she know how you feel about foreigners?



     


    She most certainly does. And I don't have anything in particular against foreigners, in most cases. I'm just pointing out that people who speak ridiculously have no right to complain about the accuracy of a computerized speech recognition program. If I tried to give commands in Spanish or Japanese or German, that would most definitely sound pretty ridiculous too.

     0Likes 0Dislikes 0Informatives
  • Reply 80 of 143
    hellacoolhellacool Posts: 759member


    When I was in highschool many moons ago, understanding the teacher 86% of the time and only getting 62% of my answer's right was a big fat F, of course I just wrote BETA on the top of my tests and all was forgiven.

     0Likes 0Dislikes 0Informatives
Sign In or Register to comment.