Street test measures Siri comprehension at 83%, accuracy at 62%
While the convenience of Apple's Siri comes with searching by voice, a new test has found that Apple's personal assistant is not yet a reliable enough search engine to entirely replace text input with Google.
The closer look at Siri and its capabilities was conducted by analyst Gene Munster of Piper Jaffray. To test Siri, Munster and his team gave the software 800 queries in a quiet room, with no headphones, and 800 queries on a busy Minneapolis street through Skull Candy microphone headphones.
The test measured Siri comprehension on the street at 83 percent, with accuracy at 62 percent. In the quiet room, Siri fared slightly better, comprehending 89 percent of queries and accurately answering 68 percent of answers.
Compared to searching Google, with text queries manually keyed in, Siri didn't fare as well. When Google was asked the same queries, the search engine provided an estimated 86 percent answer accuracy.
"We believe the most likely substitute for Siri is Google text input on the iPhone, not the Google voice search app," Munster explained of his methodology to AppleInsider.
Of the 83 percent of queries Siri could comprehend, 21 percent were answered inaccurately. Some of the questions that stumped Siri were:
His testing also found that Siri relies on Google for 60 percent of its answers in iOS 5. But this fall, with the launch of iOS 6, Munster projects that Siri's reliance on Google will drop to 48 percent, replacing the search giant with Apple's own solutions for mapping and sports scores.
"While Siri is two-plus years behind Google today, we are optimistic about its future," Munster wrote. "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google."
The closer look at Siri and its capabilities was conducted by analyst Gene Munster of Piper Jaffray. To test Siri, Munster and his team gave the software 800 queries in a quiet room, with no headphones, and 800 queries on a busy Minneapolis street through Skull Candy microphone headphones.
The test measured Siri comprehension on the street at 83 percent, with accuracy at 62 percent. In the quiet room, Siri fared slightly better, comprehending 89 percent of queries and accurately answering 68 percent of answers.
Compared to searching Google, with text queries manually keyed in, Siri didn't fare as well. When Google was asked the same queries, the search engine provided an estimated 86 percent answer accuracy.
"We believe the most likely substitute for Siri is Google text input on the iPhone, not the Google voice search app," Munster explained of his methodology to AppleInsider.
Of the 83 percent of queries Siri could comprehend, 21 percent were answered inaccurately. Some of the questions that stumped Siri were:
- "What team does Peyton Manning play for?" Siri would respond with the answer to the previous query.
- "Where is Elvis buried?" Siri looked for a person named "Elvis Buried."
- "Where am I?" Sometimes a map pin would be dropped in the wrong place.
- "When did the movie Cinderella come out?" Siri would attempt to find current theater movie times via Yelp.
- "How do I get from Boston to New York?" Siri responded: "I can only give directions from your current location. I can't give you directions to a place you are not in."
His testing also found that Siri relies on Google for 60 percent of its answers in iOS 5. But this fall, with the launch of iOS 6, Munster projects that Siri's reliance on Google will drop to 48 percent, replacing the search giant with Apple's own solutions for mapping and sports scores.
"While Siri is two-plus years behind Google today, we are optimistic about its future," Munster wrote. "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google."
Comments
Dictated flawlessly on an iPad.
The only real indication is what the millions of Siri users think about the product. One survey (using a random sample which should be representative) found that 96% if iPhone 4S users were somewhat satisfied or very satisfied with their phone and when asked about the best thing about the phone, 50% said 'Siri'.
http://www.uswitch.com/mobiles/news/2011/12/iphone_4s_siri_score_big_for_customer_satisfaction/
Or, you could do a side by side comparison using the same speakers and the same conditions. Siri had 96% accuracy compared to Google Voice at 93%:
http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/
Of course, the real benefit is not just the accuracy, it's what you can do with it. If you're driving, there's some value in "Siri, make an appointment with John for 9 am next Wednesday". Even if you have to repeat it, it's better than having to rely on your memory until you get home or trying to manually enter the appointment while driving.
Did I understand this correctly - Munster directly compared Siri to ... Google search by text input?
Siri gave me directions from Boston to New York and I live in the San Francisco Bay area.
Dictated on my iPad
Yes, Munster also reported that having sex with a real person is 12% better than with an inflatable doll... But, 35% less reliable.
Quote:
Originally Posted by ankleskater
Did I understand this correctly - Munster directly compared Siri to ... Google search by text input?
I've no doubt that Mr. Munster will do a side-by-side test between Siri and Google Now once it's in release, as will hundreds of others. I don't expect miracles but would not be at all surprised that Google would offer the more accurate results between the two. I'd personally expect comprehension to be a draw. Both iOS and Android have pretty good voice recognition already.
By the way, isn't Munster generally a pro-Apple guy? Certainly no sign that he cares much for Google AFAIK. I'd be surprised if his tests were intended to make Siri look bad.
Only 12% better? I'll stick with Fluffy Farah, thank you.
In the case of Google, forever???
Can Google do Chinese Siri equivalent? Curious if Google does this kind of stuff, honestly I couldn't be a55ed looking it up. Gatorguy?
Edit: Suffice to say Mandarin and Cantonese Siri will be nothing to sneeze at.
It's most certainly not more than beta at this stage. I seriously wish Apple would stop those stupid celebrity ads that appear to overstate its capabilities.
Quote:
Originally Posted by sr2012
Can Google do Chinese Siri equivalent? If my Mandarin was not too far off base I asked it what was its name in Mandarin, it said in Mandarin "My name is Siri, but you already knew that". Curious if Google does this kind of stuff, honestly I couldn't be a55ed looking it up. Gatorguy?
OK it does work, this time Mandarin Siri just said "my name is Siri". Can't post screenshot because it is iOS6 so I don't want to violate the NDA stuff ~too much~.
I've not used it, but a quick Google search shows that yes, it's supported...
along with 21 other languages (if you count Pig-Latin
http://support.google.com/mobile/bin/answer.py?hl=en&answer=170820
Quote:
Originally Posted by Wurm5150
Did they test it using iOS 5 or iOS 6?
iOS5 according to the article which states that iOS6 will bring improvements to Siri.
Quote:
Originally Posted by sr2012
Only 12% better? I'll stick with Fluffy Farah, thank you.
In the case of Google, forever???
I think it depends as much on the ability of the sexor as the sexee.
Yep. Not very intelligent.
Aside from the fact that he's comparing a text input to voice recognition, there are no controls or no way to compare how Siri stacks up against the competition.
Fortunately, we don't have to rely on guesses from Apple haters and google shills.
One side by side comparison shows the opposite - Siri is more accurate than Google Voice:
http://blog.thearorareport.com/2011/11/14/aaple-siri-voice-recgnition-trumps-android-voice/
Quote:
Originally Posted by Rogifan
How long is a service allows to be called beta? As long as its not working well we just call it beta to excuse away its problems.
It's "allowed" to be called beta as long as it's in beta. And it's still in beta because they haven't worked out the problems completely. As soon as they're worked out, it will probably be out of beta. Make sense?
"Stacks" up... Nudge nudge, wink wink.
ROFL. Pig Latin. Fastest growing language in 2012.
Spanish (USA) and local food searches (Australia), so many other countries... Apple doing Maps and turn-by-turn themselves...
So basically, yeah, like Tim Cook said, they are "doubling down" on Siri. Google might be able to keep pace, maybe not.
But iOS6 along with a super sexy iPhone (imagine iPhone 4S but with only as thick as the stainless steel band) is going to be... Pretty darn significant.