Street test measures Siri comprehension at 83%, accuracy at 62%

135678

Comments

  • Reply 41 of 143
    newbeenewbee Posts: 2,055member

    Quote:

    Originally Posted by DaHarder View Post





    Then Apple should stop touting it as a primary 'feature' of the iPhone 4S in those ads/TV commercials until the 'feature' is complete i.e. Out of Beta.


    DaHarder, I almost feel sorry for you. You keep trying to "badmouth" Apple .... but as each week passes and Apple keeps growing it's popularity, it's getting "DaHarder" and "DaHarder" for you to say anything at all .... that is, if you want to be truthful or relevant. Lucky for you that doesn't seem to be a priority. So sad.

     0Likes 0Dislikes 0Informatives
  • Reply 42 of 143
    mac_dogmac_dog Posts: 1,106member


    siri is still in beta, so i'd say the figures are about right.


     


    the one thing siri can do that google search can't is correct a majority of a speaker's pronunciation of their native language (when available in their country).


    in this case, the pronunciation of the american english language has—not unlike it's citizenry—gotten fat, lazy and sloppy.


     


    to me, that's priceless.

     0Likes 0Dislikes 0Informatives
  • Reply 43 of 143
    macbook promacbook pro Posts: 1,605member
    rogifan wrote: »
    How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.

    We should revisit the "how long is a service allowed to be called beta" issue in another 4 1/2 years to see if Siri defeats Google's longest running beta record.
     0Likes 0Dislikes 0Informatives
  • Reply 44 of 143
    johndoe98johndoe98 Posts: 278member

    Quote:

    Originally Posted by ankleskater View Post


    Did I understand this correctly - Munster directly compared Siri to ... Google search by text input? 

     



     


    Not only that, if you really wanted to see it's accuracy, you would have asked Siri to Google those queries, that way we can compared exactly how Siri compares to typed-input. The problem with this study is it is comparing two things at once. Siri's accuracy/comprehension, and Wolfram Alpha vs Google. How about making some of those queries specific to Wolfram's strengths and then we'll see just how good Google's searching is. Ask it some Math.

     0Likes 0Dislikes 0Informatives
  • Reply 45 of 143
    mstonemstone Posts: 11,510member

    Quote:

    Originally Posted by jragosta View Post





    I guess critical thinking isn't your thing. I didn't object because of the result but because it was a stupid test. There were major flaws in his test:

    1. Comparing a voice recognition system to manual text entry is inane.

    2. His experiment is not reproducible because we don't know what queries he used and there is no control.

    Once again, you obviously never took a critical thinking course. Nothing I said is grounds for any kind of lawsuit. What I said was:

    "If someone uses a sample of native English speakers vs. a bunch of people just off the boat who learned English from a guidebook, the answer will be different."

    That is a true, completely factual statement. I didn't say that anyone working for Munster fell into either category. I simply pointed out that voice recognition systems will depend on how similar the voice is to the average speaker for that language. If you want to do a real comparison, use the same speakers reading the same text to Siri and Google's voice recognition - and Siri comes out ahead

    As to 'racist and xenophobic', that's one of the more bizarre comments you've ever made (and that's saying a lot). Simply stating that a voice recognition system will have more trouble understanding someone who doesn't speak good English is xenophobic? Pretty bizarre conclusion.


    As obnoxious as your rant is, you do give rise to a valid question. If an English as a second language person was capable of communicating in English with a native speaker, then shouldn't one goal for Siri be the ability to also understand those same speech patterns? That to me would be a better test of the system's sophistication. We certainly have a wide range of regional accents even among native English speakers.

     0Likes 0Dislikes 0Informatives
  • Reply 46 of 143
    macbook promacbook pro Posts: 1,605member
    jragosta wrote: »
    The problem is that this is a limited test and can not be easily verified. If someone uses a sample of native English speakers vs. a bunch of people just off the boat who learned English from a guidebook, the answer will be different.
    The only real indication is what the millions of Siri users think about the product. One survey (using a random sample which should be representative) found that 96% if iPhone 4S users were somewhat satisfied or very satisfied with their phone and when asked about the best thing about the phone, 50% said 'Siri'.
    http://www.uswitch.com/mobiles/news/2011/12/iphone_4s_siri_score_big_for_customer_satisfaction/
    Or, you could do a side by side comparison using the same speakers and the same conditions. Siri had 96% accuracy compared to Google Voice at 93%:
    http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/
    Of course, the real benefit is not just the accuracy, it's what you can do with it. If you're driving, there's some value in "Siri, make an appointment with John for 9 am next Wednesday". Even if you have to repeat it, it's better than having to rely on your memory until you get home or trying to manually enter the appointment while driving.

    I seriously doubt Google will be able to provide a eyes-free, hands-free fully voice interactive experience for a long time. What many people don't realize is that Siri is the result of a decade of development by some of the greatest minds at some of the best technology schools in the United States at a total estimated cost of USD $200 million. A significant component in the development of Siri was a semantic web interface for scheduling events which is what enables Siri to communicate with various services such as Apple apps (Calendars, Contacts, Messages, Reminders) and especially external services such as movietickets.com, Wolfram Alpha, Yelp!.
     0Likes 0Dislikes 0Informatives
  • Reply 47 of 143
    jmc54jmc54 Posts: 207member

    Quote:

    Originally Posted by robogobo View Post


    It's "allowed" to be called beta as long as it's in beta.  And it's still in beta because they haven't worked out the problems completely.  As soon as they're worked out, it will probably be out of beta.  Make sense?



    google mail was in beta for what seemed like forever!

     0Likes 0Dislikes 0Informatives
  • Reply 48 of 143
    gatorguygatorguy Posts: 24,740member

    Quote:

    Originally Posted by jragosta View Post





    ROTFLMAO.

    Only in your world would a paid advertisement with no methodology or controls be a valid response to an independent side-by-side comparison with appropriate controls.

    You know you've lost when you have to resort to an advertisement to make a claim that a product is better.


    Dude, don't start out the day failing to read before posting, particularly when it's your own link. Only in your world could your article "proving" Siri more accurate than Google Voice Actions be misconstrued as a scientific study "with appropriate controls".  The article itself plainly says: 


     


    "The study is small and not scientific, therefor no quantitative conclusions should be drawn


    http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/


     


    That's your link!! Please be careful today. It's a long fall off your high-horse.

     0Likes 0Dislikes 0Informatives
  • Reply 49 of 143
    jragostajragosta Posts: 10,473member
    mstone wrote: »
    As obnoxious as your rant is, you do give rise to a valid question. If an English as a second language person was capable of communicating in English with a native speaker, then shouldn't one goal for Siri be the ability to also understand those same speech patterns? That to me would be a better test of the system's sophistication. We certainly have a wide range of regional accents even among native English speakers.

    While that is certainly the goal, any thinking person realizes that a voice recognition system is going to fail when confronted with a speaker who is barely understandable.

    A voice recognition system starts with the average voice and works from there. As it gets better, it learns to recognize voices that differ by more and more from the average, but under any conceivable system, a voice that is too far from average is going to be harder to recognize than an average voice.
    gatorguy wrote: »
    Dude, don't start out the day failing to read before posting, particularly when it's your own link. Only in your world could your article "proving" Siri more accurate than Google Voice Actions be misconstrued as a scientific study "with appropriate controls".  The article itself plainly says: 

    [SIZE=18px]"The study is small [SIZE=20px]and not scientific[/SIZE], therefor [SIZE=20px]no quantitative conclusions should be drawn[/SIZE]" [/SIZE]
    http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/

    [SIZE=14px]That's your link!![/SIZE] Please be careful today. It's a long fall off your high-horse.

    And, yet, in the story I provided, they used multiple speakers and multiple different Android phones and read the same text to all of them (and the iPhones). So a comparison can easily be made. You may not like it and the author certainly is heading his bets, but it was a scientific survey.

    Your citation was a Motorola advertisement that said that Motorola's phone was better.

    Which do you think has more validity?
     0Likes 0Dislikes 0Informatives
  • Reply 50 of 143
    ankleskaterankleskater Posts: 1,287member

    Quote:

    Originally Posted by zeromeus View Post




    Yes!  What a stupid test!  Of course the text input would be more accurate.  This isn't exactly news.





    I wonder if he will compare it to sign language next?

     0Likes 0Dislikes 0Informatives
  • Reply 51 of 143
    jlanddjlandd Posts: 873member

    Quote:



    Originally Posted by Rogifan View Post





    How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.


     




     


    Quote:

    Originally Posted by robogobo View Post




    It's "allowed" to be called beta as long as it's in beta.  And it's still in beta because they haven't worked out the problems completely.  As soon as they're worked out, it will probably be out of beta.  Make sense?


     


     


     


    Makes plenty of sense, but it doesn't address the issue. which is not how well Siri works but rather why are people NOT expecting to be beta testers when they get an iPhone.


     


    It's all well and good that Apple still disclaims it on its Siri page on their site.   But I watched the SJ and ZD celebrity ads now just to check and there was no reference to it being in beta and no direction to go to the Apple site for clarification of its usage.  Apple is trying to play both sides of the fence, staying officially in beta while having a big public relations push with A-listers that doesn't mention it, as it has been in print ads I've seen.  


     


    Obviously they're in the legal right and are not being outright deceptive, but some of those who are making objections to how it works aren't claiming it's not beta, but how this is/isn't presented.   To have the right to have a widely released, heavily advertised function be covered by the "beta" conditions it would seem you'd need to have it disclaimed on EVERYthing you use to promote it.   If that would be unwieldy or have a negative effect then you don't get to simply leave it off.  To me THAT's the issue, not that it's beta or that it's not perfect.

     0Likes 0Dislikes 0Informatives
  • Reply 52 of 143
    gatorguygatorguy Posts: 24,740member

    Quote:

    Originally Posted by jragosta View Post



    And, yet, in the story I provided, they used multiple speakers and multiple different Android phones and read the same text to all of them (and the iPhones). So a comparison can easily be made. You may not like it and the author certainly is heading his bets, but it was a scientific survey.

    Your citation was a Motorola advertisement that said that Motorola's phone was better.

    Which do you think has more validity?


    The articles' authors (who are investment traders, possibly vested in Apple for all you know) say the study "was not scientific" and "no... conclusions should be drawn". Jragosta says the study was scientific and comparisons can easily be made. Who do you think has more validity, the guys that wrote the article or you?  I vote neither, tho the traders haven't yet proved to me that they're dishonest.  


     


    Geesh, yet another instance where Jr can't be wrong. Yet here you are...

     0Likes 0Dislikes 0Informatives
  • Reply 53 of 143
    hirohiro Posts: 2,663member

    Quote:

    Originally Posted by lightknight View Post




    So, basically since (for once) Munster has a result that doesn't proclaim Apple's dominance, his experiment is null and void? He explains how he did it, it's therefore verifiable. Moreover, your comment smells of racism and/or xenophobia. Let me highlight this for you: "bunch of people just off the boat who learned English from a guidebook". I _hope_ for you nobody on Munster's team is from foreign origin and willing to take you to court. Provided of course, they've got nothing better to do than read your comments, like me ;)



     


    Munster has always been a tool.  He is the Digitimes of US Stock analysts.  Anything that comes from him can be immediately dismissed as churn fodder.  


     


    Also take your threatening faux sense of superiority and hide it in your nether regions.  Your definition of racism/xenophobia is completely self serving and horribly misapplied.  While the post you responded to was a bit hyperbolic, it was not anything of which you describe.

     0Likes 0Dislikes 0Informatives
  • Reply 54 of 143
    jragostajragosta Posts: 10,473member
    gatorguy wrote: »
    The articles' author says the study "was not scientific" and "no... conclusions should be drawn". Jragosta says the study was scientific and comparisons can easily be made. Who do you think has more validity, the guys that wrote the article or you? Geesh, yet another instance where Jr can't be wrong. Yet here you are...

    Are you a flipping idiot or what?

    I have a PhD in science. I know what is scientific.

    Now, a scientific study may or may not have validity, but if it meets the premises of science, it is a scientific study. That is, it must have:
    1. A testable hypothesis. Check
    2. A methodology for testing the hypothesis. Check.
    3. A control. Check.
    4. Results. Check.
    5. A conclusion. Check.

    It meets the criteria. They are being careful to ensure that it's not being used as the be-all and end-all of research on the subject, but to say that it's not scientific just because it doesn't have a lot of replication is just plain wrong.

    And, no matter how you slice it, it's infinitely more valid than a Motorola advertisement saying that Motorola's phones are better than Apple's phones. That should be obvious to anyone thinking at above a 5 year old's level. Sadly, that seems to leave you out.
     0Likes 0Dislikes 0Informatives
  • Reply 55 of 143
    markbyrnmarkbyrn Posts: 662member


    I guess Siri will be in beta for years to come.


     


    lsqgia26.jpg

     0Likes 0Dislikes 0Informatives
  • Reply 56 of 143

    Quote:

    Originally Posted by jragosta View Post





    1. Comparing a voice recognition system to manual text entry is inane.


    Why is it inane? Other than you obviously not liking the result?


     


    At the end of the day, they're both search engines. Siri translates the voice input to text and then submits it to a search engine. The accuracy of those searches was vastly in favor of Google. Deal with it.


     


    Siri is a fun feature but its overall performance is lacking. It's great at the simple tasks, like setting alarms, providing directions home, voice dialing, but when you tell it to search for things it really does nose dive into uselessness.

     0Likes 0Dislikes 0Informatives
  • Reply 57 of 143
    macbook promacbook pro Posts: 1,605member
    Notably, SRI and Apple have only begun to demonstrate the currently available scope of Siri. Siri can do far more than anyone has seen in a released product today.

    Siri can do [I][B]Workflow Activity Recognition and Proactive Assistance (just like Google Now)[/B][/I] today, this simply hasn't been activated by Apple.
    Siri can [B][I]handle email meeting requests, reserve venues, and schedule events then find emails, documents, and contacts that are relevant to a given meeting automatically[/I][/B], this simply hasn't been activated by Apple.
    Siri can [I][B]automatically compose email messages and suggest attachments for the message[/B][/I] in response to scheduled events or emails, this simply hasn't been activated by Apple.
    Siri can [I][B]automatically search the Internet for contact information and additional data for your contacts[/B][/I], this simply hasn't been activated by Apple. For example, if you have a meeting with Susan Jeffries who is a high school friend you haven't seen for five years Siri can tell you she is married with two children (with their names and birthdays) and employed by Bank of America.
    Siri can track your interests and [I][B]generate a user-personalized RSS feeds[/B][/I], this simply hasn't been activated by Apple.

    [B][I]Siri can learn and automate complex workflows. To understand how powerful Siri is you need to understand how transformative the computer revolution was to businesses in the 1980s.[/I][/B]

    When Apple enables Siri on the Mac platform and enables use of these features, Siri will be very disruptive to business in a way that almost can't be understated. The problem is that this will be very disruptive to Apple as well since the flood of demand would be overwhelming. One might be forgiven for believing this might even be the beginning of the demise of the technology juggernaut, Microsoft.
     0Likes 0Dislikes 0Informatives
  • Reply 58 of 143
    mstonemstone Posts: 11,510member

    Quote:

    Originally Posted by jragosta View Post





    While that is certainly the goal, any thinking person realizes that a voice recognition system is going to fail when confronted with a speaker who is barely understandable.

    A voice recognition system starts with the average voice and works from there. As it gets better, it learns to recognize voices that differ by more and more from the average, but under any conceivable system, a voice that is too far from average is going to be harder to recognize than an average voice.


    I've been using Google's voice recognition for quite a while and it works pretty well in both English and Spanish. I just started using Apple's and it seems to be on par with English but so far Spanish is not supported. I'm looking forward to the release later this year.

     0Likes 0Dislikes 0Informatives
  • Reply 59 of 143
    kent909kent909 Posts: 731member


    I think this is pretty impressive. My wife does not comprehend me and she is rarely right.  

     0Likes 0Dislikes 0Informatives
  • Reply 60 of 143

    Quote:

    Originally Posted by MacBook Pro View Post



    Notably, SRI and Apple have only begun to demonstrate the currently available scope of Siri. Siri can do far more than anyone has seen in a released product today.

    Siri can do Workflow Activity Recognition and Proactive Assistance (just like Google Now) today, this simply hasn't been activated by Apple.

    Siri can handle email meeting requests, reserves venues, and schedules events today then find emails, documents, and contacts that are relevant to a given meeting automatically, this simply hasn't been activated by Apple.

    Siri can automatically compose email messages and suggest attachments for the message in response to scheduled events or emails, this simply hasn't been activated by Apple.

    Siri can automatically search the Internet for contact information and additional data for your contacts, this simply hasn't been activated by Apple. For example, if you have a meeting with Susan Jeffries who is a high school friend you haven't seen for five years Siri can tell you she is married with two children (with their names and birthdays) and employed by Bank of America.

    Siri can track your interests and generate a user-personalized RSS feeds, this simply hasn't been activated by Apple.

    Siri can learn and automate complex workflows. To understand how powerful Siri is you need to understand how transformative the computer revolution was to businesses in the 1980s.


     


     


    Got a link?  I know Siri was developed as a US  Government program, but I've never heard of these abilities.

     0Likes 0Dislikes 0Informatives
Sign In or Register to comment.