Street test measures Siri comprehension at 83%, accuracy at 62%

Posted:
in iPhone edited January 2014
While the convenience of Apple's Siri comes with searching by voice, a new test has found that Apple's personal assistant is not yet a reliable enough search engine to entirely replace text input with Google.

The closer look at Siri and its capabilities was conducted by analyst Gene Munster of Piper Jaffray. To test Siri, Munster and his team gave the software 800 queries in a quiet room, with no headphones, and 800 queries on a busy Minneapolis street through Skull Candy microphone headphones.

The test measured Siri comprehension on the street at 83 percent, with accuracy at 62 percent. In the quiet room, Siri fared slightly better, comprehending 89 percent of queries and accurately answering 68 percent of answers.

Compared to searching Google, with text queries manually keyed in, Siri didn't fare as well. When Google was asked the same queries, the search engine provided an estimated 86 percent answer accuracy.

"We believe the most likely substitute for Siri is Google text input on the iPhone, not the Google voice search app," Munster explained of his methodology to AppleInsider.



Of the 83 percent of queries Siri could comprehend, 21 percent were answered inaccurately. Some of the questions that stumped Siri were:
  • "What team does Peyton Manning play for?" Siri would respond with the answer to the previous query.
  • "Where is Elvis buried?" Siri looked for a person named "Elvis Buried."
  • "Where am I?" Sometimes a map pin would be dropped in the wrong place.
  • "When did the movie Cinderella come out?" Siri would attempt to find current theater movie times via Yelp.
  • "How do I get from Boston to New York?" Siri responded: "I can only give directions from your current location. I can't give you directions to a place you are not in."
In Munster's test, he gave Google an A+ for comprehension and B+ for accuracy, while Apple's Siri was graded with a B for comprehension and a D for accuracy.

His testing also found that Siri relies on Google for 60 percent of its answers in iOS 5. But this fall, with the launch of iOS 6, Munster projects that Siri's reliance on Google will drop to 48 percent, replacing the search giant with Apple's own solutions for mapping and sports scores.

"While Siri is two-plus years behind Google today, we are optimistic about its future," Munster wrote. "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google."
«1345678

Comments

  • Reply 1 of 143
    macbook promacbook pro Posts: 1,605member
    This is an excellent argument for Apple purchasing Yahoo!, in my opinion. Yahoo! has very robust services in virtually every area needed to make Siri a serious threat to Google search on Apple devices.
  • Reply 2 of 143
    psych_guypsych_guy Posts: 486member
    That's why Siri is said to still be in beta.

    Dictated flawlessly on an iPad.
  • Reply 3 of 143
    jragostajragosta Posts: 10,473member
    The problem is that this is a limited test and can not be easily verified. If someone uses a sample of native English speakers vs. a bunch of people just off the boat who learned English from a guidebook, the answer will be different.

    The only real indication is what the millions of Siri users think about the product. One survey (using a random sample which should be representative) found that 96% if iPhone 4S users were somewhat satisfied or very satisfied with their phone and when asked about the best thing about the phone, 50% said 'Siri'.
    http://www.uswitch.com/mobiles/news/2011/12/iphone_4s_siri_score_big_for_customer_satisfaction/


    Or, you could do a side by side comparison using the same speakers and the same conditions. Siri had 96% accuracy compared to Google Voice at 93%:
    http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/

    Of course, the real benefit is not just the accuracy, it's what you can do with it. If you're driving, there's some value in "Siri, make an appointment with John for 9 am next Wednesday". Even if you have to repeat it, it's better than having to rely on your memory until you get home or trying to manually enter the appointment while driving.
  • Reply 4 of 143
    ankleskaterankleskater Posts: 1,287member


    Did I understand this correctly - Munster directly compared Siri to ... Google search by text input? 

     

  • Reply 5 of 143
    dick applebaumdick applebaum Posts: 12,527member


    Siri gave me directions from Boston to New York and I live in the San Francisco Bay area.


     


    Dictated on my iPad

  • Reply 6 of 143
    dick applebaumdick applebaum Posts: 12,527member


    Yes, Munster also reported that having sex with a real person is 12% better than with an inflatable doll... But, 35% less reliable.

  • Reply 7 of 143
    rogifanrogifan Posts: 10,669member
    psych_guy wrote: »
    That's why Siri is said to still be in beta.
    Dictated flawlessly on an iPad.
    How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.
  • Reply 8 of 143
    gatorguygatorguy Posts: 24,176member

    Quote:

    Originally Posted by ankleskater View Post


    Did I understand this correctly - Munster directly compared Siri to ... Google search by text input? 

     



    I've no doubt that Mr. Munster will do a side-by-side test between Siri and Google Now once it's in release, as will hundreds of others. I don't expect miracles but would not be at all surprised that Google would offer the more accurate results between the two. I'd personally expect comprehension to be a draw. Both iOS and Android have pretty good voice recognition already.


     


    By the way, isn't Munster generally a pro-Apple guy? Certainly no sign that he cares much for Google AFAIK. I'd be surprised if his tests were intended to make Siri look bad. 

  • Reply 9 of 143
    sr2012sr2012 Posts: 896member
    Statistics can be used to prove anything. 80% of people know this.
  • Reply 10 of 143
    sr2012sr2012 Posts: 896member
    Yes, Munster also reported that having sex with a real person is 12% better than with an inflatable doll.

    Only 12% better? I'll stick with Fluffy Farah, thank you.
    rogifan wrote: »
    How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.

    In the case of Google, forever???
  • Reply 11 of 143
    sr2012sr2012 Posts: 896member
    Siri gave me directions from Boston to New York and I live in the San Francisco Bay area.

    Dictated on my iPad

    Can Google do Chinese Siri equivalent? Curious if Google does this kind of stuff, honestly I couldn't be a55ed looking it up. Gatorguy?

    Edit: Suffice to say Mandarin and Cantonese Siri will be nothing to sneeze at.
  • Reply 12 of 143
    ghostface147ghostface147 Posts: 1,629member
    Works fine for me. I'd say a little more accurate than 68%.
  • Reply 13 of 143
    anantksundaramanantksundaram Posts: 20,403member
    The best one can say about Siri is that it's fairly decent, and work in progress. It has a ways to go before I would reliably depend on it.

    It's most certainly not more than beta at this stage. I seriously wish Apple would stop those stupid celebrity ads that appear to overstate its capabilities.
  • Reply 14 of 143
    gatorguygatorguy Posts: 24,176member

    Quote:

    Originally Posted by sr2012 View Post





    Can Google do Chinese Siri equivalent? If my Mandarin was not too far off base I asked it what was its name in Mandarin, it said in Mandarin "My name is Siri, but you already knew that". Curious if Google does this kind of stuff, honestly I couldn't be a55ed looking it up. Gatorguy?

    OK it does work, this time Mandarin Siri just said "my name is Siri". Can't post screenshot because it is iOS6 so I don't want to violate the NDA stuff ~too much~.


    I've not used it, but a quick Google search shows that yes, it's supported...


    along with 21 other languages (if you count Pig-Latin image). 


    http://support.google.com/mobile/bin/answer.py?hl=en&answer=170820

  • Reply 15 of 143
    wurm5150wurm5150 Posts: 763member
    Did they test it using iOS 5 or iOS 6?
  • Reply 16 of 143
    gatorguygatorguy Posts: 24,176member

    Quote:

    Originally Posted by Wurm5150 View Post



    Did they test it using iOS 5 or iOS 6?


    iOS5 according to the article which states that iOS6 will bring improvements to Siri.

  • Reply 17 of 143
    dick applebaumdick applebaum Posts: 12,527member

    Quote:

    Originally Posted by sr2012 View Post





    Only 12% better? I'll stick with Fluffy Farah, thank you.

    In the case of Google, forever???


     


    I think it depends  as much on the ability of the sexor as the sexee.

  • Reply 18 of 143
    jragostajragosta Posts: 10,473member
    Did I understand this correctly - Munster directly compared Siri to ... Google search by text input? 

    Yep. Not very intelligent.

    Aside from the fact that he's comparing a text input to voice recognition, there are no controls or no way to compare how Siri stacks up against the competition.
    gatorguy wrote: »
    I've no doubt that Mr. Munster will do a side-by-side test between Siri and Google Now once it's in release, as will hundreds of others. I don't expect miracles but would not be at all surprised that Google would offer the more accurate results between the two. I'd personally expect comprehension to be a draw. Both iOS and Android have pretty good voice recognition already.

    Fortunately, we don't have to rely on guesses from Apple haters and google shills.

    One side by side comparison shows the opposite - Siri is more accurate than Google Voice:
    http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/
  • Reply 19 of 143
    robogoborobogobo Posts: 378member

    Quote:

    Originally Posted by Rogifan View Post





    How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.


    It's "allowed" to be called beta as long as it's in beta.  And it's still in beta because they haven't worked out the problems completely.  As soon as they're worked out, it will probably be out of beta.  Make sense?

  • Reply 20 of 143
    sr2012sr2012 Posts: 896member
    jragosta wrote: »
    Yep. Not very intelligent.
    Aside from the fact that he's comparing a text input to voice recognition, there are no controls or no way to compare how Siri stacks up against the competition.
    Fortunately, we don't have to rely on guesses from Apple haters and google shills.
    One side by side comparison shows the opposite - Siri is more accurate than Google Voice:
    http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/

    "Stacks" up... Nudge nudge, wink wink.

    gatorguy wrote: »
    I've not used it, but a quick Google search shows that yes, it's supported...
    along with 21 other languages (if you count Pig-Latin :lol: ). 
    http://support.google.com/mobile/bin/answer.py?hl=en&answer=170820

    ROFL. Pig Latin. Fastest growing language in 2012.

    Spanish (USA) and local food searches (Australia), so many other countries... Apple doing Maps and turn-by-turn themselves...

    So basically, yeah, like Tim Cook said, they are "doubling down" on Siri. Google might be able to keep pace, maybe not.

    But iOS6 along with a super sexy iPhone (imagine iPhone 4S but with only as thick as the stainless steel band) is going to be... Pretty darn significant.
Sign In or Register to comment.