or Connect
AppleInsider › Forums › Mobile › iPhone › Tests find Apple's Siri improving, but Google Now voice search slightly better
New Posts  All Forums:Forum Nav:

Tests find Apple's Siri improving, but Google Now voice search slightly better

post #1 of 42
Thread Starter 
The comprehension abilities of Apple's voice-driven personal assistant Siri continue to get better, but new testing shows that using Google Now for searching on Android has surpassed it in terms of accuracy when answering queries.




Analyst Gene Munster of Piper Jaffray did a side-by-side comparison of Apple's Siri and Google Now this month, and the results of those tests were shared with AppleInsider in a new research note on Tuesday. He found that the Android voice search function correctly answered questions 84 percent of the time, barely edging out Siri, which was 82 percent correct.

Both platforms have seen major improvements in comprehension levels. In particular, Siri was found this month to correctly interpret questions 96 percent of the time, up from an 88 percent accuracy rate in tests conducted in December of 2012.

But there's still a disparity between correctly hearing the question and correctly answering it. When Siri heard the question correctly, it still provided the wrong answer 16 percent of the time.




Last December, Munster ranked Apple's Siri with a grade of "C+", which at the time was even with his assessment of Google Now. But in his latest July rankings, while Siri improved to a "B-", Google's search was given a slightly better grade of "B".

Munster's tests also found that Siri has continued to reduce its reliance on Google, and answered just 3 percent of questions using data from the search giant. That's down considerably from the 27 percent for which it relied on Google in December of 2012.

Apple has moved away from Google Maps, and also uses Microsoft's Bing as its default search provider for traditional Web content. Apple Maps has taken over not only for Google, but also for most Yelp content.

"We believe Siri will continue increasing the number of queries it can answer without consulting outside sources," the analyst wrote. "This is important because if Siri consistently directs users to other search engines, they will be more likely to simply use Google/another search engine instead of going through Siri."




Most notably for Munster, Siri has gained the ability to answer questions using two sources at once. In one example, he asked "Where is Mt. St. Helen?", and the personal assistant provided information from both Apple Maps and Wikipedia.

In the past, Google Now has had an advantage over Siri because it uses Google Search, Google Maps, and Google Play together to provide an integrated and comprehensive answer to queries," Munster said. Siri is catching up on this front."

In terms of improvements that Apple could make, Munster noted that Siri does not sort search results by price or hours of availability. Google Now, however, gives users the ability to filter results based on price, location, rating and hours.

Siri does give the ability to sort results based on Yelp ratings, or distance.

Munster's testing was done in both a controlled environment, simulating minimal background noise that a user might experience when using their smartphone indoors, as well as an uncontrolled environment. In the uncontrolled test, background noise was kept at about 80 decibels.
post #2 of 42
I could see with the latest IBM deal that Watson (GoAT Jeopardy champ) has joined team Apple, and its technology might start coming to Siri some day.
post #3 of 42
The last graphic seems to give Google Now the edge in most cases, but with the caveat that the data only reflects situations where the question was heard correctly. Now if you look at the July 2014 data about correct comprehension in an uncontrolled environment for both services (the only thing most of us experience....) the comprehension rate of Siri is significantly higher. Google Now misunderstands the question 71% more often than Siri. How is that not considered huge and why isn't it a main point of the article?
post #4 of 42

Think the Google Now Report Card needs an error correction - Heard Incorrectly in a Controlled Environment 100% of the time?!

censored

Reply

censored

Reply
post #5 of 42
I wonder when Siri will get offline processing. That would be nice.
post #6 of 42

I'm not sure I understand the top row of the table.

 

Google Now:

93% correctly heard * 86% answered correctly when correctly heard = 79.98% correctly answered overall

 

Apple Siri:

96% correctly heard * 84% correctly answered when correctly heard = 80.64% correctly answered overall

 

So Siri, despite being a bit flaky on answering questions correctly, still wins overall.  And it's a bigger advantage to Siri if you only use the un controlled environment percentages.

 

How have Piper Jaffray calculated their "Ability to answer questions"?

censored

Reply

censored

Reply
post #7 of 42
Gene will do or say anything for attention. It's sad, really.

Proud AAPL stock owner.

 

GOA

Reply

Proud AAPL stock owner.

 

GOA

Reply
post #8 of 42

Siri generally works well for me. However, I really wish it would work without the internet for tasks such as texting, placing a call or checking my calendar. As long as the person is in my contacts or the event is in my calendar, Siri should not have to go out to perform these tasks.

post #9 of 42
Quote:
Originally Posted by mike1 View Post

Siri generally works well for me. However, I really wish it would work without the internet for tasks such as texting, placing a call or checking my calendar. As long as the person is in my contacts or the event is in my calendar, Siri should not have to go out to perform these tasks.

I agree. However, from what I understand, it uses the network/Apple Siri server etc to analyze your voice to understand the request correctly, does not matter if it's for local command or not. But till that day...
Hokey religions and ancient weapons are no match for a good blaster by your side, kid.
Reply
Hokey religions and ancient weapons are no match for a good blaster by your side, kid.
Reply
post #10 of 42

Can someone explain to me why Google Now went from a 4%'heard incorrectly' in December 2013 to 100% 'heard incorrectly' in July 2014? What's the point of releasing these stats unless you double-check for typos? Worthless.

post #11 of 42

I'm not sure Gene is the right person to do this comparison. Testing with a single voice in a single language is too small of a sample size, although it is impressive that he got so many "heard correctly" results. You'd think having ones head up ones ass would tend to muffle the voice.

Life is too short to drink bad coffee.

Reply

Life is too short to drink bad coffee.

Reply
post #12 of 42
Gene Muenster is the Michael Pachter of the tech world.

Too many Apple products to list...Long on AAPL, so take what I say with a bucket of salt.

You are only relevant...if your customers love you.

Reply

Too many Apple products to list...Long on AAPL, so take what I say with a bucket of salt.

You are only relevant...if your customers love you.

Reply
post #13 of 42

I hope Apple does not rest on the fact they are slightly better than Google. Siri needs huge improvements on ability to assist, not just recognition. Siri still can't add/update contacts and is not even close to going past 1 level of thought. 

post #14 of 42
Quote:
Originally Posted by Sigma902 View Post

I could see with the latest IBM deal that Watson (GoAT Jeopardy champ) has joined team Apple, and its technology might start coming to Siri some day.

doubtful.  at least not soon.  I don't think sharing that sort of IP was mentioned anywhere.   

post #15 of 42
Quote:
Originally Posted by Crowley View Post

Think the Google Now Report Card needs an error correction - Heard Incorrectly in a Controlled Environment 100% of the time?!
Looks like you just discovered DED's next editorial, "Study discovers Google Now incorrect 100% of the time."
"Proof is irrelevant" - Solipsism
Reply
"Proof is irrelevant" - Solipsism
Reply
post #16 of 42
Quote:
Originally Posted by Rogifan View Post

I wonder when Siri will get offline processing. That would be nice.

 

I don't think I have enough free space on my phone to store the entire Bing database, complete Yelp library and maps of the entire world.

 

;)

Lorin Schultz (formerly V5V)

Audio Engineer

V5V Digital Media, Vancouver, BC Canada

Reply

Lorin Schultz (formerly V5V)

Audio Engineer

V5V Digital Media, Vancouver, BC Canada

Reply
post #17 of 42
Now open for business: the Gene Munster Product Evaluation Labs.

"Apple should pull the plug on the iPhone."

John C. Dvorak, 2007
Reply

"Apple should pull the plug on the iPhone."

John C. Dvorak, 2007
Reply
post #18 of 42
Quote:
Originally Posted by Rogifan View Post

I wonder when Siri will get offline processing. That would be nice.

It's a matter time. Eventually, IBM Watson will be considered as quaint and outdated as the UNIVAC. When that day comes, our smartphones will be able to play Jeopardy! while in airplane mode (assuming wifi phobia on air travel is still in existence). Oh, it's coming. And we will still whine that a 64 Exabyte iPhone costs $100 more than the 32... 1wink.gif

"Apple should pull the plug on the iPhone."

John C. Dvorak, 2007
Reply

"Apple should pull the plug on the iPhone."

John C. Dvorak, 2007
Reply
post #19 of 42
Quote:
Originally Posted by Lorin Schultz View Post
 

 

I don't think I have enough free space on my phone to store the entire Bing database, complete Yelp library and maps of the entire world.

 

;)

Sure, but the speak to text part could feasibly be on-device, and then the comprehension engine could deal with local actions, such as "play music", "send an iMessage", "open app" etc, without recourse to the internet.

censored

Reply

censored

Reply
post #20 of 42
Quote:
Originally Posted by Rogifan View Post

I wonder when Siri will get offline processing. That would be nice.

I think Microsoft's Cortana also offers some offline processing too.

EDIT:
http://www.tomshardware.com/news/microsoft-cortana-unique-features,26506.html
Edited by Gatorguy - 7/22/14 at 9:21am
melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #21 of 42

I wonder if Gene:

 

Recorded his queries and played them back to both phones. If not, subtle differences in the voice of the person doing the speaking could alter the results.

 

Recorded the background noise used in the uncontrolled environment and played the same recording to both phones.

 

Placed the background noise AND the query in the same recording so each query as heard by the phone had the same exact background noise and spoken query.

 

Was in a soundproof room while doing this test. Or, at least in a quiet environment and had people and things in the same physical location in the room while the test was underway.

 

All of these things don't make a LOT of difference, but they do some, and taken together could sway his results.

post #22 of 42
1) I am surprised to see Siri doing so well, especially where it bests Google Now in key areas. Perhaps the real-time-ish transcription will go a long way to alter that perception of it being slower than Google Now.

2) I'm surprised that iOS 6 and iOS 7 results for accuracy are so different when I assume this is all handled on the server. I wonder what devices they used for each.

This bot has been removed from circulation due to a malfunctioning morality chip.

Reply

This bot has been removed from circulation due to a malfunctioning morality chip.

Reply
post #23 of 42
Quote:
Originally Posted by SolipsismX View Post

1) I am surprised to see Siri doing so well, especially where it bests Google Now in key areas. Perhaps the real-time-ish transcription will go a long way to alter that perception of it being slower than Google Now.

2) I'm surprised that iOS 6 and iOS 7 results for accuracy are so different when I assume this is all handled on the server. I wonder what devices they used for each.

Without Gene being a bit more detailed it's impossible to know how reliable and repeatable the testing was. Siri could actually be returning even more dependable results than Google Now ( which would not be totally unexpected IMHO) but you can't tell from the limited information provided here. FWIW that's just what Munster found back in 2012 and again in December last year, Siri offering better or at least equivalent results to Google Now. If the roles were reversed in the latest tests with Google Now performing better it would be a mild surprise.

EDIT: Added clarifier.
Edited by Gatorguy - 7/22/14 at 10:21am
melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #24 of 42
Quote:
Originally Posted by Gatorguy View Post

Without Gene being a bit more detailed it's impossible to know how reliable and repeatable the testing was. Siri could actually be returning even more dependable results than Google Now ( which would be totally unexpected IMHO) but you can't tell from the limited information provided here. FWIW that's just what Munster found back in 2012 and again in December last year, Siri offering better or at least equivalent results to Google Now. If the roles were reversed in the latest tests with Google Now performing better it would be a mild surprise.

Good points. What we really need are the same set of robust questions and commands asked for the various tests (although perhaps not written out for us to see so neither company can dope their results artificially) and completed by many individuals using both devices each time so we can an idea for accuracy with various accents, dialects, and other speech patterns. I'd also like you see various languages detailed.

This bot has been removed from circulation due to a malfunctioning morality chip.

Reply

This bot has been removed from circulation due to a malfunctioning morality chip.

Reply
post #25 of 42
I wish Siri had the ability to do simple things offline... like set a timer, set an alarm, or launch an app.

Not sure why those simple tasks an online connection. Is Siri a completely dumb client (basically a microphone to the cloud)? Is there no local processing going on?
post #26 of 42

I noticed a major degradation in Siri with iOS7.  Accuracy has been for shit.  I have to correct much of it, even though I speak loud enough and slowly enough.  

post #27 of 42
Originally Posted by Psych_guy View Post
I noticed a major degradation in Siri with iOS7.

 

I agree. It seems to have gotten worse lately. Specifically, if I ask it a question about any famous figure, Siri tells me she can't find him/her in my contacts list. Also, any foreign sounding word seems to be a lose for Siri. I then try it on Google, and get much better question comprehension, though often inaccurate results.

post #28 of 42
Quote:
Originally Posted by TokyoJimu View Post
 

 

I agree. It seems to have gotten worse lately. Specifically, if I ask it a question about any famous figure, Siri tells me she can't find him/her in my contacts list. Also, any foreign sounding word seems to be a lose for Siri. I then try it on Google, and get much better question comprehension, though often inaccurate results.


I have seen a few problems lately as well as Siri being unavailable every once in awhile. Trying the question again seemy  to get the right answer and if she is off line trying again right away usually finds her back online. My theory is that Apple is making some big data base changes and upgrades in anticipation of iOS 8. I didn't have these problems before the iOS 8 beta went live. IMO Apple is working on something big in their data centers that is causing some hiccups right now. Should it be happening? Absolutely not! But it might be worth it in the long run. I could be completely wrong about this. I hope I'm not. I have no proof and I expect that there will be others here that will knock my head against the wall with their comments.

post #29 of 42
Quote:
Originally Posted by Crowley View Post
 

Sure, but the speak to text part could feasibly be on-device, and then the comprehension engine could deal with local actions, such as "play music", "send an iMessage", "open app" etc, without recourse to the internet.


Since most iPhones are sold with 16GB storage the space used for proper voice recognition would take up too much storage space in that configuration. The integration with other Siri commands that require on-line mode would also be troublesome. Some options that Apple could integrate are make the base system 32GB or I think maybe a better solution could be a voice processor integrated into the system similar to how the M7 motion processor was added. Of course that would require dealing with battery, space, heat, etc. Since I'm not an electrical engineer I'm not even sure if a voice processor is possible at this point in time. Any engineers have input?

post #30 of 42
Quote:
Originally Posted by mnbob1 View Post


Since most iPhones are sold with 16GB storage the space used for proper voice recognition would take up too much storage space in that configuration. The integration with other Siri commands that require on-line mode would also be troublesome. Some options that Apple could integrate are make the base system 32GB or I think maybe a better solution could be a voice processor integrated into the system similar to how the M7 motion processor was added. Of course that would require dealing with battery, space, heat, etc. Since I'm not an electrical engineer I'm not even sure if a voice processor is possible at this point in time. Any engineers have input?

I don't know that it's the problem you imagine it to be. Google Now (and Microsoft Cortana too I think) includes off-line voice recognition and the ability to do perform some searches and find some information by voice alone and without an internet connection. At least with Google Android it works with hardware much less stellar than what Apple uses.
Edited by Gatorguy - 7/22/14 at 12:35pm
melior diabolus quem scies
Reply
melior diabolus quem scies
Reply
post #31 of 42
Quote:
Originally Posted by mnbob1 View Post
 

Since most iPhones are sold with 16GB storage the space used for proper voice recognition would take up too much storage space in that configuration. 

Don't see why.  Google Now and Microsoft Cortana seem to do just fine, and handsets for Android and Windows Phone 8 have the same sort of capacities.

censored

Reply

censored

Reply
post #32 of 42

I know this article is about voice recognition. I have only used Siri but I have been told that Google Now has the ability to answer questions in series. Such as: Who's pitching for the Twins today? Yohan Pino What are his stats?

Siri will answer a question but doesn't recognize the answer to the previous question.

Is this accurate about Google Now? How well does it actually work? Just wondering. Sorry for going off topic.

post #33 of 42
Apple should bring highlights to spotlight since in iOS 8 it's finally becoming on par with Google now but is not easy enough to discover IMO. From the lock screen searching for a place is now super easy as well as calling directly without having the number. But they should include a 'search with maps' option.
post #34 of 42
Quote:
Originally Posted by mnbob1 View Post
 

I know this article is about voice recognition. I have only used Siri but I have been told that Google Now has the ability to answer questions in series. Such as: Who's pitching for the Twins today? Yohan Pino What are his stats?

Siri will answer a question but doesn't recognize the answer to the previous question.

Is this accurate about Google Now? How well does it actually work? Just wondering. Sorry for going off topic.

 

When asked, "Do the Twins play today?"  I got a verbal answer of "The Twin are playing the Indians today at 7:10 PM"  I follow with, "Who's pitching?" and it did a Google search for "who's pitching for the twins today" (no verbal answer).

 

I also asked "How tall is Barack Obama?" and got a correct answer spoken back.  I followed up with, "How old is he?" and also got a correct answer spoken back.

"Proof is irrelevant" - Solipsism
Reply
"Proof is irrelevant" - Solipsism
Reply
post #35 of 42
Quote:
Originally Posted by mnbob1 View Post
 

I know this article is about voice recognition. I have only used Siri but I have been told that Google Now has the ability to answer questions in series. Such as: Who's pitching for the Twins today? Yohan Pino What are his stats?

Siri will answer a question but doesn't recognize the answer to the previous question.

Is this accurate about Google Now? How well does it actually work? Just wondering. Sorry for going off topic.

 

 

Quote:
Originally Posted by DroidFTW View Post
 

 

When asked, "Do the Twins play today?"  I got a verbal answer of "The Twin are playing the Indians today at 7:10 PM"  I follow with, "Who's pitching?" and it did a Google search for "who's pitching for the twins today" (no verbal answer).

 

I also asked "How tall is Barack Obama?" and got a correct answer spoken back.  I followed up with, "How old is he?" and also got a correct answer spoken back.

 

It can use contextual reference points as DroidFTW stated.  Here are two quick examples.

 

Q - "How tall is Barack Obama?"  A - "Barack Obama is 6'1" tall."

Q - "What about Michelle?"  A - "Michelle Obama is 5'11" tall."

Q - "And how old is she?"  A - "Michelle Obama is 50 years old."

 

Q - "How far away is Jupiter?"  A - "...365 million miles..."

Q - "No, Jupiter FL!"  A - "The drive from your location is 163.2 miles."

Q - "And Tampa?"  A - "The drive from your location is 92.7 miles."

post #36 of 42
Can people write articles that focus on how people might really use these tools? It was only until now that you could even use Google Now from a locked screen. Why no mention of that?
OK, why not try and ask Google Now to give you directions to somewhere? 90% of the time it searches the web. It doesn't offer you choices like Siri at all. I use both iPhone and Android ALL THE TIME and I can tell you that Siri is still much more useful on a day to day basis for hands free use.

OK try this. Say send a text to "will be there soon".
Siri will READ THE TEXT out loud - like you really want.

Google Now will NOT say anything but "do you want to send it"? Yeah, how useful is that when you are hands free.

It's just really irritating about how ridiculous these writers are. There are a lot of GREAT things in Android, but Siri (at the moment) is much more useful for hands free use.
post #37 of 42
Sorry, Apple.

I have a iPhone 5S and Google Now is a lot better that Siri.
Why?
Google Now can understand Portuguese BR. Siri doesn't.
post #38 of 42
Originally Posted by Overlord View Post
I have a iPhone 5S and Google Now is a lot better that Siri. Why? Google Now can understand Portuguese BR. Siri doesn't.


So, no, it’s not actually better.

post #39 of 42
My experience with Siri is that it's still very flaky and for me anyway Google Now still works much better. I'm glad to read here that it's improving though. But it's going to have to get a lot better before I go back to it.
post #40 of 42
Quote:
Originally Posted by SolipsismX View Post

1) I am surprised to see Siri doing so well, especially where it bests Google Now in key areas. Perhaps the real-time-ish transcription will go a long way to alter that perception of it being slower than Google Now.

2) I'm surprised that iOS 6 and iOS 7 results for accuracy are so different when I assume this is all handled on the server. I wonder what devices they used for each.

Re iOS differences: better microphone or sound processing?
"If the young are not initiated into the village, they will burn it down just to feel its warmth."
- African proverb
Reply
"If the young are not initiated into the village, they will burn it down just to feel its warmth."
- African proverb
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: iPhone
  • Tests find Apple's Siri improving, but Google Now voice search slightly better
AppleInsider › Forums › Mobile › iPhone › Tests find Apple's Siri improving, but Google Now voice search slightly better