Tests find Apple's Siri improving, but Google Now voice search slightly better
The comprehension abilities of Apple's voice-driven personal assistant Siri continue to get better, but new testing shows that using Google Now for searching on Android has surpassed it in terms of accuracy when answering queries.
Analyst Gene Munster of Piper Jaffray did a side-by-side comparison of Apple's Siri and Google Now this month, and the results of those tests were shared with AppleInsider in a new research note on Tuesday. He found that the Android voice search function correctly answered questions 84 percent of the time, barely edging out Siri, which was 82 percent correct.
Both platforms have seen major improvements in comprehension levels. In particular, Siri was found this month to correctly interpret questions 96 percent of the time, up from an 88 percent accuracy rate in tests conducted in December of 2012.
But there's still a disparity between correctly hearing the question and correctly answering it. When Siri heard the question correctly, it still provided the wrong answer 16 percent of the time.
Last December, Munster ranked Apple's Siri with a grade of "C+", which at the time was even with his assessment of Google Now. But in his latest July rankings, while Siri improved to a "B-", Google's search was given a slightly better grade of "B".
Munster's tests also found that Siri has continued to reduce its reliance on Google, and answered just 3 percent of questions using data from the search giant. That's down considerably from the 27 percent for which it relied on Google in December of 2012.
Apple has moved away from Google Maps, and also uses Microsoft's Bing as its default search provider for traditional Web content. Apple Maps has taken over not only for Google, but also for most Yelp content.
"We believe Siri will continue increasing the number of queries it can answer without consulting outside sources," the analyst wrote. "This is important because if Siri consistently directs users to other search engines, they will be more likely to simply use Google/another search engine instead of going through Siri."
Most notably for Munster, Siri has gained the ability to answer questions using two sources at once. In one example, he asked "Where is Mt. St. Helen?", and the personal assistant provided information from both Apple Maps and Wikipedia.
In the past, Google Now has had an advantage over Siri because it uses Google Search, Google Maps, and Google Play together to provide an integrated and comprehensive answer to queries," Munster said. Siri is catching up on this front."
In terms of improvements that Apple could make, Munster noted that Siri does not sort search results by price or hours of availability. Google Now, however, gives users the ability to filter results based on price, location, rating and hours.
Siri does give the ability to sort results based on Yelp ratings, or distance.
Munster's testing was done in both a controlled environment, simulating minimal background noise that a user might experience when using their smartphone indoors, as well as an uncontrolled environment. In the uncontrolled test, background noise was kept at about 80 decibels.
Analyst Gene Munster of Piper Jaffray did a side-by-side comparison of Apple's Siri and Google Now this month, and the results of those tests were shared with AppleInsider in a new research note on Tuesday. He found that the Android voice search function correctly answered questions 84 percent of the time, barely edging out Siri, which was 82 percent correct.
Both platforms have seen major improvements in comprehension levels. In particular, Siri was found this month to correctly interpret questions 96 percent of the time, up from an 88 percent accuracy rate in tests conducted in December of 2012.
But there's still a disparity between correctly hearing the question and correctly answering it. When Siri heard the question correctly, it still provided the wrong answer 16 percent of the time.
Last December, Munster ranked Apple's Siri with a grade of "C+", which at the time was even with his assessment of Google Now. But in his latest July rankings, while Siri improved to a "B-", Google's search was given a slightly better grade of "B".
Munster's tests also found that Siri has continued to reduce its reliance on Google, and answered just 3 percent of questions using data from the search giant. That's down considerably from the 27 percent for which it relied on Google in December of 2012.
Apple has moved away from Google Maps, and also uses Microsoft's Bing as its default search provider for traditional Web content. Apple Maps has taken over not only for Google, but also for most Yelp content.
"We believe Siri will continue increasing the number of queries it can answer without consulting outside sources," the analyst wrote. "This is important because if Siri consistently directs users to other search engines, they will be more likely to simply use Google/another search engine instead of going through Siri."
Most notably for Munster, Siri has gained the ability to answer questions using two sources at once. In one example, he asked "Where is Mt. St. Helen?", and the personal assistant provided information from both Apple Maps and Wikipedia.
In the past, Google Now has had an advantage over Siri because it uses Google Search, Google Maps, and Google Play together to provide an integrated and comprehensive answer to queries," Munster said. Siri is catching up on this front."
In terms of improvements that Apple could make, Munster noted that Siri does not sort search results by price or hours of availability. Google Now, however, gives users the ability to filter results based on price, location, rating and hours.
Siri does give the ability to sort results based on Yelp ratings, or distance.
Munster's testing was done in both a controlled environment, simulating minimal background noise that a user might experience when using their smartphone indoors, as well as an uncontrolled environment. In the uncontrolled test, background noise was kept at about 80 decibels.
Comments
Think the Google Now Report Card needs an error correction - Heard Incorrectly in a Controlled Environment 100% of the time?!
I'm not sure I understand the top row of the table.
Google Now:
93% correctly heard * 86% answered correctly when correctly heard = 79.98% correctly answered overall
Apple Siri:
96% correctly heard * 84% correctly answered when correctly heard = 80.64% correctly answered overall
So Siri, despite being a bit flaky on answering questions correctly, still wins overall. And it's a bigger advantage to Siri if you only use the un controlled environment percentages.
How have Piper Jaffray calculated their "Ability to answer questions"?
Siri generally works well for me. However, I really wish it would work without the internet for tasks such as texting, placing a call or checking my calendar. As long as the person is in my contacts or the event is in my calendar, Siri should not have to go out to perform these tasks.
I agree. However, from what I understand, it uses the network/Apple Siri server etc to analyze your voice to understand the request correctly, does not matter if it's for local command or not. But till that day...
Can someone explain to me why Google Now went from a 4%'heard incorrectly' in December 2013 to 100% 'heard incorrectly' in July 2014? What's the point of releasing these stats unless you double-check for typos? Worthless.
I'm not sure Gene is the right person to do this comparison. Testing with a single voice in a single language is too small of a sample size, although it is impressive that he got so many "heard correctly" results. You'd think having ones head up ones ass would tend to muffle the voice.
I hope Apple does not rest on the fact they are slightly better than Google. Siri needs huge improvements on ability to assist, not just recognition. Siri still can't add/update contacts and is not even close to going past 1 level of thought.
I could see with the latest IBM deal that Watson (GoAT Jeopardy champ) has joined team Apple, and its technology might start coming to Siri some day.
doubtful. at least not soon. I don't think sharing that sort of IP was mentioned anywhere.
I wonder when Siri will get offline processing. That would be nice.
I don't think I have enough free space on my phone to store the entire Bing database, complete Yelp library and maps of the entire world.
It's a matter time. Eventually, IBM Watson will be considered as quaint and outdated as the UNIVAC. When that day comes, our smartphones will be able to play Jeopardy! while in airplane mode (assuming wifi phobia on air travel is still in existence). Oh, it's coming. And we will still whine that a 64 Exabyte iPhone costs $100 more than the 32...
I don't think I have enough free space on my phone to store the entire Bing database, complete Yelp library and maps of the entire world.
Sure, but the speak to text part could feasibly be on-device, and then the comprehension engine could deal with local actions, such as "play music", "send an iMessage", "open app" etc, without recourse to the internet.
I think Microsoft's Cortana also offers some offline processing too.
EDIT:
http://www.tomshardware.com/news/microsoft-cortana-unique-features,26506.html
I wonder if Gene:
Recorded his queries and played them back to both phones. If not, subtle differences in the voice of the person doing the speaking could alter the results.
Recorded the background noise used in the uncontrolled environment and played the same recording to both phones.
Placed the background noise AND the query in the same recording so each query as heard by the phone had the same exact background noise and spoken query.
Was in a soundproof room while doing this test. Or, at least in a quiet environment and had people and things in the same physical location in the room while the test was underway.
All of these things don't make a LOT of difference, but they do some, and taken together could sway his results.