or Connect
AppleInsider › Forums › Mobile › iPhone › Street test measures Siri comprehension at 83%, accuracy at 62%
New Posts  All Forums:Forum Nav:

Street test measures Siri comprehension at 83%, accuracy at 62%

post #1 of 141
Thread Starter 
While the convenience of Apple's Siri comes with searching by voice, a new test has found that Apple's personal assistant is not yet a reliable enough search engine to entirely replace text input with Google.

The closer look at Siri and its capabilities was conducted by analyst Gene Munster of Piper Jaffray. To test Siri, Munster and his team gave the software 800 queries in a quiet room, with no headphones, and 800 queries on a busy Minneapolis street through Skull Candy microphone headphones.

The test measured Siri comprehension on the street at 83 percent, with accuracy at 62 percent. In the quiet room, Siri fared slightly better, comprehending 89 percent of queries and accurately answering 68 percent of answers.

Compared to searching Google, with text queries manually keyed in, Siri didn't fare as well. When Google was asked the same queries, the search engine provided an estimated 86 percent answer accuracy.

"We believe the most likely substitute for Siri is Google text input on the iPhone, not the Google voice search app," Munster explained of his methodology to AppleInsider.



Of the 83 percent of queries Siri could comprehend, 21 percent were answered inaccurately. Some of the questions that stumped Siri were:
  • "What team does Peyton Manning play for?" Siri would respond with the answer to the previous query.
  • "Where is Elvis buried?" Siri looked for a person named "Elvis Buried."
  • "Where am I?" Sometimes a map pin would be dropped in the wrong place.
  • "When did the movie Cinderella come out?" Siri would attempt to find current theater movie times via Yelp.
  • "How do I get from Boston to New York?" Siri responded: "I can only give directions from your current location. I can't give you directions to a place you are not in."

In Munster's test, he gave Google an A+ for comprehension and B+ for accuracy, while Apple's Siri was graded with a B for comprehension and a D for accuracy.

His testing also found that Siri relies on Google for 60 percent of its answers in iOS 5. But this fall, with the launch of iOS 6, Munster projects that Siri's reliance on Google will drop to 48 percent, replacing the search giant with Apple's own solutions for mapping and sports scores.

"While Siri is two-plus years behind Google today, we are optimistic about its future," Munster wrote. "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google."
post #2 of 141
This is an excellent argument for Apple purchasing Yahoo!, in my opinion. Yahoo! has very robust services in virtually every area needed to make Siri a serious threat to Google search on Apple devices.
post #3 of 141
That's why Siri is said to still be in beta.

Dictated flawlessly on an iPad.
post #4 of 141
The problem is that this is a limited test and can not be easily verified. If someone uses a sample of native English speakers vs. a bunch of people just off the boat who learned English from a guidebook, the answer will be different.

The only real indication is what the millions of Siri users think about the product. One survey (using a random sample which should be representative) found that 96% if iPhone 4S users were somewhat satisfied or very satisfied with their phone and when asked about the best thing about the phone, 50% said 'Siri'.
http://www.uswitch.com/mobiles/news/2011/12/iphone_4s_siri_score_big_for_customer_satisfaction/


Or, you could do a side by side comparison using the same speakers and the same conditions. Siri had 96% accuracy compared to Google Voice at 93%:
http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/

Of course, the real benefit is not just the accuracy, it's what you can do with it. If you're driving, there's some value in "Siri, make an appointment with John for 9 am next Wednesday". Even if you have to repeat it, it's better than having to rely on your memory until you get home or trying to manually enter the appointment while driving.
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
post #5 of 141

Did I understand this correctly - Munster directly compared Siri to ... Google search by text input? 
 

post #6 of 141

Siri gave me directions from Boston to New York and I live in the San Francisco Bay area.

 

Dictated on my iPad

"Swift generally gets you to the right way much quicker." - auxio -

"The perfect [birth]day -- A little playtime, a good poop, and a long nap." - Tomato Greeting Cards -
Reply
"Swift generally gets you to the right way much quicker." - auxio -

"The perfect [birth]day -- A little playtime, a good poop, and a long nap." - Tomato Greeting Cards -
Reply
post #7 of 141

Yes, Munster also reported that having sex with a real person is 12% better than with an inflatable doll... But, 35% less reliable.


Edited by Dick Applebaum - 6/29/12 at 6:22am
"Swift generally gets you to the right way much quicker." - auxio -

"The perfect [birth]day -- A little playtime, a good poop, and a long nap." - Tomato Greeting Cards -
Reply
"Swift generally gets you to the right way much quicker." - auxio -

"The perfect [birth]day -- A little playtime, a good poop, and a long nap." - Tomato Greeting Cards -
Reply
post #8 of 141
Quote:
Originally Posted by Psych_guy View Post

That's why Siri is said to still be in beta.
Dictated flawlessly on an iPad.
How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.
post #9 of 141
Quote:
Originally Posted by ankleskater View Post

Did I understand this correctly - Munster directly compared Siri to ... Google search by text input? 
 

I've no doubt that Mr. Munster will do a side-by-side test between Siri and Google Now once it's in release, as will hundreds of others. I don't expect miracles but would not be at all surprised that Google would offer the more accurate results between the two. I'd personally expect comprehension to be a draw. Both iOS and Android have pretty good voice recognition already.

 

By the way, isn't Munster generally a pro-Apple guy? Certainly no sign that he cares much for Google AFAIK. I'd be surprised if his tests were intended to make Siri look bad. 

melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #10 of 141
Statistics can be used to prove anything. 80% of people know this.
post #11 of 141
Quote:
Originally Posted by Dick Applebaum View Post

Yes, Munster also reported that having sex with a real person is 12% better than with an inflatable doll.

Only 12% better? I'll stick with Fluffy Farah, thank you.
Quote:
Originally Posted by Rogifan View Post

How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.

In the case of Google, forever???
post #12 of 141
Quote:
Originally Posted by Dick Applebaum View Post

Siri gave me directions from Boston to New York and I live in the San Francisco Bay area.

Dictated on my iPad

Can Google do Chinese Siri equivalent? Curious if Google does this kind of stuff, honestly I couldn't be a55ed looking it up. Gatorguy?

Edit: Suffice to say Mandarin and Cantonese Siri will be nothing to sneeze at.
Edited by sr2012 - 6/29/12 at 6:59am
post #13 of 141
Works fine for me. I'd say a little more accurate than 68%.
post #14 of 141
The best one can say about Siri is that it's fairly decent, and work in progress. It has a ways to go before I would reliably depend on it.

It's most certainly not more than beta at this stage. I seriously wish Apple would stop those stupid celebrity ads that appear to overstate its capabilities.
post #15 of 141
Quote:
Originally Posted by sr2012 View Post


Can Google do Chinese Siri equivalent? If my Mandarin was not too far off base I asked it what was its name in Mandarin, it said in Mandarin "My name is Siri, but you already knew that". Curious if Google does this kind of stuff, honestly I couldn't be a55ed looking it up. Gatorguy?
OK it does work, this time Mandarin Siri just said "my name is Siri". Can't post screenshot because it is iOS6 so I don't want to violate the NDA stuff ~too much~.

I've not used it, but a quick Google search shows that yes, it's supported...

along with 21 other languages (if you count Pig-Latin lol.gif). 

http://support.google.com/mobile/bin/answer.py?hl=en&answer=170820

melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #16 of 141
Did they test it using iOS 5 or iOS 6?
post #17 of 141
Quote:
Originally Posted by Wurm5150 View Post

Did they test it using iOS 5 or iOS 6?

iOS5 according to the article which states that iOS6 will bring improvements to Siri.

melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #18 of 141
Quote:
Originally Posted by sr2012 View Post


Only 12% better? I'll stick with Fluffy Farah, thank you.
In the case of Google, forever???

 

I think it depends  as much on the ability of the sexor as the sexee.

"Swift generally gets you to the right way much quicker." - auxio -

"The perfect [birth]day -- A little playtime, a good poop, and a long nap." - Tomato Greeting Cards -
Reply
"Swift generally gets you to the right way much quicker." - auxio -

"The perfect [birth]day -- A little playtime, a good poop, and a long nap." - Tomato Greeting Cards -
Reply
post #19 of 141
Quote:
Originally Posted by ankleskater View Post

Did I understand this correctly - Munster directly compared Siri to ... Google search by text input? 

Yep. Not very intelligent.

Aside from the fact that he's comparing a text input to voice recognition, there are no controls or no way to compare how Siri stacks up against the competition.
Quote:
Originally Posted by Gatorguy View Post

I've no doubt that Mr. Munster will do a side-by-side test between Siri and Google Now once it's in release, as will hundreds of others. I don't expect miracles but would not be at all surprised that Google would offer the more accurate results between the two. I'd personally expect comprehension to be a draw. Both iOS and Android have pretty good voice recognition already.

Fortunately, we don't have to rely on guesses from Apple haters and google shills.

One side by side comparison shows the opposite - Siri is more accurate than Google Voice:
http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
post #20 of 141
Quote:
Originally Posted by Rogifan View Post


How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.

It's "allowed" to be called beta as long as it's in beta.  And it's still in beta because they haven't worked out the problems completely.  As soon as they're worked out, it will probably be out of beta.  Make sense?

post #21 of 141
Quote:
Originally Posted by jragosta View Post

Yep. Not very intelligent.
Aside from the fact that he's comparing a text input to voice recognition, there are no controls or no way to compare how Siri stacks up against the competition.
Fortunately, we don't have to rely on guesses from Apple haters and google shills.
One side by side comparison shows the opposite - Siri is more accurate than Google Voice:
http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/

"Stacks" up... Nudge nudge, wink wink.

Quote:
Originally Posted by Gatorguy View Post

I've not used it, but a quick Google search shows that yes, it's supported...
along with 21 other languages (if you count Pig-Latin lol.gif ). 
http://support.google.com/mobile/bin/answer.py?hl=en&answer=170820

ROFL. Pig Latin. Fastest growing language in 2012.

Spanish (USA) and local food searches (Australia), so many other countries... Apple doing Maps and turn-by-turn themselves...

So basically, yeah, like Tim Cook said, they are "doubling down" on Siri. Google might be able to keep pace, maybe not.

But iOS6 along with a super sexy iPhone (imagine iPhone 4S but with only as thick as the stainless steel band) is going to be... Pretty darn significant.
Edited by sr2012 - 6/29/12 at 6:58am
post #22 of 141
Quote:
Originally Posted by jragosta View Post

Fortunately, we don't have to rely on guesses from Apple haters and google shills.
One side by side comparison shows the opposite - Siri is more accurate than Google Voice:
http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/

I believe I suggested Munster would do a future side-by-side with Google Now, not Google Voice. Read more carefully please. So yes, even you still have to rely on guesses for now on how they'll compare.

 

BTW, this is a much more recent test putting the two head-to-head. One test doesn't prove anything of course, whether this one or Munsters.

http://www.youtube.com/watch?v=kDsOtdRtG0Q&feature=player_embedded

melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #23 of 141
Quote:
Originally Posted by sr2012 View Post

Statistics can be used to prove anything. 80% of people know this.


Made-up statistics don't prove anything. 99% of people agree with me.

Social Capitalist, dreamer and wise enough to know I'm never going to grow up anyway... so not trying anymore.

 

http://m.ign.com/articles/2014/07/16/7-high-school-girls-are-kickstarting-their-awa...

Reply

Social Capitalist, dreamer and wise enough to know I'm never going to grow up anyway... so not trying anymore.

 

http://m.ign.com/articles/2014/07/16/7-high-school-girls-are-kickstarting-their-awa...

Reply
post #24 of 141
Quote:
Originally Posted by Dick Applebaum View Post

I think it depends  as much on the ability of the sexor as the sexee.

Well, me and Fluffy Farah, it's 100% better than real-life then... LOL
post #25 of 141
Quote:
Originally Posted by jragosta View Post

The problem is that this is a limited test and can not be easily verified. If someone uses a sample of native English speakers vs. a bunch of people just off the boat who learned English from a guidebook, the answer will be different.
The only real indication is what the millions of Siri users think about the product. One survey (using a random sample which should be representative) found that 96% if iPhone 4S users were somewhat satisfied or very satisfied with their phone and when asked about the best thing about the phone, 50% said 'Siri'.
http://www.uswitch.com/mobiles/news/2011/12/iphone_4s_siri_score_big_for_customer_satisfaction/
Or, you could do a side by side comparison using the same speakers and the same conditions. Siri had 96% accuracy compared to Google Voice at 93%:
http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/
Of course, the real benefit is not just the accuracy, it's what you can do with it. If you're driving, there's some value in "Siri, make an appointment with John for 9 am next Wednesday". Even if you have to repeat it, it's better than having to rely on your memory until you get home or trying to manually enter the appointment while driving.


So, basically since (for once) Munster has a result that doesn't proclaim Apple's dominance, his experiment is null and void? He explains how he did it, it's therefore verifiable. Moreover, your comment smells of racism and/or xenophobia. Let me highlight this for you: "bunch of people just off the boat who learned English from a guidebook". I _hope_ for you nobody on Munster's team is from foreign origin and willing to take you to court. Provided of course, they've got nothing better to do than read your comments, like me ;)

Social Capitalist, dreamer and wise enough to know I'm never going to grow up anyway... so not trying anymore.

 

http://m.ign.com/articles/2014/07/16/7-high-school-girls-are-kickstarting-their-awa...

Reply

Social Capitalist, dreamer and wise enough to know I'm never going to grow up anyway... so not trying anymore.

 

http://m.ign.com/articles/2014/07/16/7-high-school-girls-are-kickstarting-their-awa...

Reply
post #26 of 141
Quote:
Originally Posted by Gatorguy View Post

I believe I suggested Munster would do a future side-by-side with Google Now, not Google Voice. Read more carefully please. So yes, even you still have to rely on guesses for now on how they'll compare.

BTW, this is a much more recent test putting the two head-to-head. One test doesn't prove anything of course, whether this one or Munsters.
http://www.youtube.com/watch?v=kDsOtdRtG0Q&feature=player_embedded

Yes, but this does point out the difference in philosophy. Go to the section where he asks it to set an alarm. Google Voice displays a page where you have to manually enter the information. Siri asks you to tell it what time to set the alarm and so on. If I'm driving or my hands are full, there's a world of difference.

The huge functionality difference is far more important than the 0.1 second time difference he keeps bragging about.

But, nonetheless, that comparison does nothing to answer the question of accuracy which is what this thread and the post you responded to was all about. Read the link I provided which shows that Siri is more accurate than the existing Google product. Since neither iOS 6.0 or Google Voice is currently available to users, we have to rely on existing products.
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
post #27 of 141
Quote:
Originally Posted by sr2012 View Post

But iOS6 along with a super sexy iPhone (imagine iPhone 4S but with only as thick as the stainless steel band) is going to be... Pretty darn significant.

If they add the expected larger 4"+ display along with LTE support, I completely agree. The next iPhone is probably a killer. 

melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #28 of 141
Quote:
Originally Posted by robogobo View Post

It's "allowed" to be called beta as long as it's in beta.  And it's still in beta because they haven't worked out the problems completely.  As soon as they're worked out, it will probably be out of beta.  Make sense?

Then Apple should stop touting it as a primary 'feature' of the iPhone 4S in those ads/TV commercials until the 'feature' is complete i.e. Out of Beta.
"Why iPhone"... Hmmm?
Reply
"Why iPhone"... Hmmm?
Reply
post #29 of 141
Quote:
Originally Posted by lightknight View Post


So, basically since (for once) Munster has a result that doesn't proclaim Apple's dominance, his experiment is null and void?

I guess critical thinking isn't your thing. I didn't object because of the result but because it was a stupid test. There were major flaws in his test:
1. Comparing a voice recognition system to manual text entry is inane.
2. His experiment is not reproducible because we don't know what queries he used and there is no control.
Quote:
Originally Posted by lightknight View Post

He explains how he did it, it's therefore verifiable. Moreover, your comment smells of racism and/or xenophobia. Let me highlight this for you: "bunch of people just off the boat who learned English from a guidebook". I _hope_ for you nobody on Munster's team is from foreign origin and willing to take you to court. Provided of course, they've got nothing better to do than read your comments, like me 1wink.gif

Once again, you obviously never took a critical thinking course. Nothing I said is grounds for any kind of lawsuit. What I said was:
"If someone uses a sample of native English speakers vs. a bunch of people just off the boat who learned English from a guidebook, the answer will be different."

That is a true, completely factual statement. I didn't say that anyone working for Munster fell into either category. I simply pointed out that voice recognition systems will depend on how similar the voice is to the average speaker for that language. If you want to do a real comparison, use the same speakers reading the same text to Siri and Google's voice recognition - and Siri comes out ahead

As to 'racist and xenophobic', that's one of the more bizarre comments you've ever made (and that's saying a lot). Simply stating that a voice recognition system will have more trouble understanding someone who doesn't speak good English is xenophobic? Pretty bizarre conclusion.
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
post #30 of 141
Quote:
Originally Posted by robogobo View Post

It's "allowed" to be called beta as long as it's in beta.  And it's still in beta because they haven't worked out the problems completely.  As soon as they're worked out, it will probably be out of beta.  Make sense?
Well you have to admit Apple is touting it as a major feature of the iPhone. I mean how many companies advertise on beta product/service?
post #31 of 141
Quote:
Originally Posted by Rogifan View Post

Well you have to admit Apple is touting it as a major feature of the iPhone. I mean how many companies advertise on beta product/service?

Google. Or maybe you've all forgotten Gmail.

Originally Posted by Marvin

The only thing more insecure than Android’s OS is its userbase.
Reply

Originally Posted by Marvin

The only thing more insecure than Android’s OS is its userbase.
Reply
post #32 of 141
Quote:
Originally Posted by jragosta View Post


Yep. Not very intelligent.
Aside from the fact that he's comparing a text input to voice recognition, there are no controls or no way to compare how Siri stacks up against the competition.
Fortunately, we don't have to rely on guesses from Apple haters and google shills.
One side by side comparison shows the opposite - Siri is more accurate than Google Voice:
http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/


Not just unintelligent. Totally misleading. Why not compare Siri to talking to a human? Why not compare Siri to training a dog? While we have come to expect silliness from professional analysts, this is lowering the bar again. Of course, Munster will become "the man" if and when Apple releases a flat panel TV, because he predicts its emergence every year.

post #33 of 141
Quote:
Originally Posted by jragosta View Post


Yes, but this does point out the difference in philosophy. Go to the section where he asks it to set an alarm. Google Voice displays a page where you have to manually enter the information. Siri asks you to tell it what time to set the alarm and so on. If I'm driving or my hands are full, there's a world of difference.
The huge functionality difference is far more important than the 0.1 second time difference he keeps bragging about.
But, nonetheless, that comparison does nothing to answer the question of accuracy which is what this thread and the post you responded to was all about. Read the link I provided which shows that Siri is more accurate than the existing Google product. Since neither iOS 6.0 or Google Voice is currently available to users, we have to rely on existing products.

And this link says just the opposite:

http://www.huffingtonpost.com/2012/02/27/motorola-siri-android-iphone_n_1304984.html

 

One, or even a couple of tests is not definitive proof. Also as you mentioned, some users might be better served by Google Voice Actions which has mor feature control functions, while Siri may fit the bill jsut fine for others. There's no clear "better" solution regardless of any current testing

melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #34 of 141
Quote:
Originally Posted by Gatorguy View Post

I've no doubt that Mr. Munster will do a side-by-side test between Siri and Google Now once it's in release, as will hundreds of others. I don't expect miracles but would not be at all surprised that Google would offer the more accurate results between the two. I'd personally expect comprehension to be a draw. Both iOS and Android have pretty good voice recognition already.

 

Siri isn't voice recognition, it's natural language processing, so, the state of Android's voice recognition (a relatively trivial problem at this point) isn't really relevant. But, thanks for the shill talk.

 

By the way, isn't Munster generally a pro-Apple guy? Certainly no sign that he cares much for Google AFAIK. I'd be surprised if his tests were intended to make Siri look bad.

 

No, he's an analyst who makes a lot of incorrect predictions about what Apple and AAPL will do. With Munster, the issue isn't malice, but rather whether he know what he's doing. But, it's amusing that you try to paint him as pro-Apple, as if you don't know exactly who he is and what he does.

post #35 of 141
Quote:
Originally Posted by ankleskater View Post

Did I understand this correctly - Munster directly compared Siri to ... Google search by text input? 
 


Yes!  What a stupid test!  Of course the text input would be more accurate.  This isn't exactly news.

post #36 of 141
Quote:
Originally Posted by Tallest Skil View Post


Google. Or maybe you've all forgotten Gmail.

That's exactly what I was about to say, but I decided to search 'gmai' before posting.  

 

TS, of all the tech sites that I follow and see you on, I must say I usually agree with your arguments.  The problem is that people like us think before we act and as a result, we become outnumbered by people that don't and have no idea what they're talking about.

 

Real life Idiocracy..

www.imdb.com/title/tt0387808/

post #37 of 141
Quote:
Originally Posted by Gatorguy View Post

And this link says just the opposite:
http://www.huffingtonpost.com/2012/02/27/motorola-siri-android-iphone_n_1304984.html

One, or even a couple of tests is not definitive proof. Also as you mentioned, some users might be better served by Google Voice Actions which has mor feature control functions, while Siri may fit the bill jsut fine for others. There's no clear "better" solution regardless of any current testing

ROTFLMAO.

Only in your world would a paid advertisement with no methodology or controls be a valid response to an independent side-by-side comparison with appropriate controls.

You know you've lost when you have to resort to an advertisement to make a claim that a product is better.
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
post #38 of 141

In my tests, the only thing that Siri gets consistently right with a 100% accuracy is setting timers and alarms. 

 

In fact, I haven't used my alarm clock since getting the 4S.

post #39 of 141
Quote:
Originally Posted by jragosta View Post


I guess critical thinking isn't your thing. 

Once again, you obviously never took a critical thinking course.
that's one of the more bizarre comments you've ever made (and that's saying a lot).
 

 

 

This stuff is completely uncalled for.

post #40 of 141
Quote:
Originally Posted by JerrySwitched26 View Post


This stuff is completely uncalled for.

Why? He posted inane personal attacks which had no basis in reality and I called him on it.
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
"I'm way over my head when it comes to technical issues like this"
Gatorguy 5/31/13
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: iPhone
AppleInsider › Forums › Mobile › iPhone › Street test measures Siri comprehension at 83%, accuracy at 62%