Tests find Apple's Siri improving, but Google Now voice search slightly better

Posted:
in iPhone edited October 2014
The comprehension abilities of Apple's voice-driven personal assistant Siri continue to get better, but new testing shows that using Google Now for searching on Android has surpassed it in terms of accuracy when answering queries.




Analyst Gene Munster of Piper Jaffray did a side-by-side comparison of Apple's Siri and Google Now this month, and the results of those tests were shared with AppleInsider in a new research note on Tuesday. He found that the Android voice search function correctly answered questions 84 percent of the time, barely edging out Siri, which was 82 percent correct.

Both platforms have seen major improvements in comprehension levels. In particular, Siri was found this month to correctly interpret questions 96 percent of the time, up from an 88 percent accuracy rate in tests conducted in December of 2012.

But there's still a disparity between correctly hearing the question and correctly answering it. When Siri heard the question correctly, it still provided the wrong answer 16 percent of the time.




Last December, Munster ranked Apple's Siri with a grade of "C+", which at the time was even with his assessment of Google Now. But in his latest July rankings, while Siri improved to a "B-", Google's search was given a slightly better grade of "B".

Munster's tests also found that Siri has continued to reduce its reliance on Google, and answered just 3 percent of questions using data from the search giant. That's down considerably from the 27 percent for which it relied on Google in December of 2012.

Apple has moved away from Google Maps, and also uses Microsoft's Bing as its default search provider for traditional Web content. Apple Maps has taken over not only for Google, but also for most Yelp content.

"We believe Siri will continue increasing the number of queries it can answer without consulting outside sources," the analyst wrote. "This is important because if Siri consistently directs users to other search engines, they will be more likely to simply use Google/another search engine instead of going through Siri."




Most notably for Munster, Siri has gained the ability to answer questions using two sources at once. In one example, he asked "Where is Mt. St. Helen?", and the personal assistant provided information from both Apple Maps and Wikipedia.

In the past, Google Now has had an advantage over Siri because it uses Google Search, Google Maps, and Google Play together to provide an integrated and comprehensive answer to queries," Munster said. Siri is catching up on this front."

In terms of improvements that Apple could make, Munster noted that Siri does not sort search results by price or hours of availability. Google Now, however, gives users the ability to filter results based on price, location, rating and hours.

Siri does give the ability to sort results based on Yelp ratings, or distance.

Munster's testing was done in both a controlled environment, simulating minimal background noise that a user might experience when using their smartphone indoors, as well as an uncontrolled environment. In the uncontrolled test, background noise was kept at about 80 decibels.
«13

Comments

  • Reply 1 of 41
    I could see with the latest IBM deal that Watson (GoAT Jeopardy champ) has joined team Apple, and its technology might start coming to Siri some day.
  • Reply 2 of 41
    The last graphic seems to give Google Now the edge in most cases, but with the caveat that the data only reflects situations where the question was heard correctly. Now if you look at the July 2014 data about correct comprehension in an uncontrolled environment for both services (the only thing most of us experience....) the comprehension rate of Siri is significantly higher. Google Now misunderstands the question 71% more often than Siri. How is that not considered huge and why isn't it a main point of the article?
  • Reply 3 of 41
    crowleycrowley Posts: 6,056member

    Think the Google Now Report Card needs an error correction - Heard Incorrectly in a Controlled Environment 100% of the time?!

  • Reply 4 of 41
    rogifanrogifan Posts: 10,669member
    I wonder when Siri will get offline processing. That would be nice.
  • Reply 5 of 41
    crowleycrowley Posts: 6,056member

    I'm not sure I understand the top row of the table.

     

    Google Now:

    93% correctly heard * 86% answered correctly when correctly heard = 79.98% correctly answered overall

     

    Apple Siri:

    96% correctly heard * 84% correctly answered when correctly heard = 80.64% correctly answered overall

     

    So Siri, despite being a bit flaky on answering questions correctly, still wins overall.  And it's a bigger advantage to Siri if you only use the un controlled environment percentages.

     

    How have Piper Jaffray calculated their "Ability to answer questions"?

  • Reply 6 of 41
    SpamSandwichSpamSandwich Posts: 31,491member
    Gene will do or say anything for attention. It's sad, really.
  • Reply 7 of 41
    mike1mike1 Posts: 1,941member

    Siri generally works well for me. However, I really wish it would work without the internet for tasks such as texting, placing a call or checking my calendar. As long as the person is in my contacts or the event is in my calendar, Siri should not have to go out to perform these tasks.

  • Reply 8 of 41
    boeyc15boeyc15 Posts: 986member
    mike1 wrote: »
    Siri generally works well for me. However, I really wish it would work without the internet for tasks such as texting, placing a call or checking my calendar. As long as the person is in my contacts or the event is in my calendar, Siri should not have to go out to perform these tasks.

    I agree. However, from what I understand, it uses the network/Apple Siri server etc to analyze your voice to understand the request correctly, does not matter if it's for local command or not. But till that day...
  • Reply 9 of 41
    wigbywigby Posts: 692member

    Can someone explain to me why Google Now went from a 4%'heard incorrectly' in December 2013 to 100% 'heard incorrectly' in July 2014? What's the point of releasing these stats unless you double-check for typos? Worthless.

  • Reply 10 of 41
    mstonemstone Posts: 11,510member

    I'm not sure Gene is the right person to do this comparison. Testing with a single voice in a single language is too small of a sample size, although it is impressive that he got so many "heard correctly" results. You'd think having ones head up ones ass would tend to muffle the voice.

  • Reply 11 of 41
    Gene Muenster is the Michael Pachter of the tech world.
  • Reply 12 of 41
    richard getzrichard getz Posts: 1,142member

    I hope Apple does not rest on the fact they are slightly better than Google. Siri needs huge improvements on ability to assist, not just recognition. Siri still can't add/update contacts and is not even close to going past 1 level of thought. 

  • Reply 13 of 41
    Quote:

    Originally Posted by Sigma902 View Post



    I could see with the latest IBM deal that Watson (GoAT Jeopardy champ) has joined team Apple, and its technology might start coming to Siri some day.

    doubtful.  at least not soon.  I don't think sharing that sort of IP was mentioned anywhere.   

  • Reply 14 of 41
    droidftwdroidftw Posts: 1,009member
    crowley wrote: »
    Think the Google Now Report Card needs an error correction - Heard Incorrectly in a Controlled Environment 100% of the time?!
    Looks like you just discovered DED's next editorial, "Study discovers Google Now incorrect 100% of the time."
  • Reply 15 of 41
    Quote:

    Originally Posted by Rogifan View Post



    I wonder when Siri will get offline processing. That would be nice.

     

    I don't think I have enough free space on my phone to store the entire Bing database, complete Yelp library and maps of the entire world.

     

    ;)

  • Reply 16 of 41
    Now open for business: the Gene Munster Product Evaluation Labs.
  • Reply 17 of 41
    rogifan wrote: »
    I wonder when Siri will get offline processing. That would be nice.

    It's a matter time. Eventually, IBM Watson will be considered as quaint and outdated as the UNIVAC. When that day comes, our smartphones will be able to play Jeopardy! while in airplane mode (assuming wifi phobia on air travel is still in existence). Oh, it's coming. And we will still whine that a 64 Exabyte iPhone costs $100 more than the 32... ;)
  • Reply 18 of 41
    crowleycrowley Posts: 6,056member
    Quote:
    Originally Posted by Lorin Schultz View Post

     

     

    I don't think I have enough free space on my phone to store the entire Bing database, complete Yelp library and maps of the entire world.

     

    ;)


    Sure, but the speak to text part could feasibly be on-device, and then the comprehension engine could deal with local actions, such as "play music", "send an iMessage", "open app" etc, without recourse to the internet.

  • Reply 19 of 41
    gatorguygatorguy Posts: 21,255member
    rogifan wrote: »
    I wonder when Siri will get offline processing. That would be nice.

    I think Microsoft's Cortana also offers some offline processing too.

    EDIT:
    http://www.tomshardware.com/news/microsoft-cortana-unique-features,26506.html
  • Reply 20 of 41
    wingswings Posts: 261member

    I wonder if Gene:

     

    Recorded his queries and played them back to both phones. If not, subtle differences in the voice of the person doing the speaking could alter the results.

     

    Recorded the background noise used in the uncontrolled environment and played the same recording to both phones.

     

    Placed the background noise AND the query in the same recording so each query as heard by the phone had the same exact background noise and spoken query.

     

    Was in a soundproof room while doing this test. Or, at least in a quiet environment and had people and things in the same physical location in the room while the test was underway.

     

    All of these things don't make a LOT of difference, but they do some, and taken together could sway his results.

Sign In or Register to comment.