Tests find iOS 7 update improves Apple's Siri, but Google Now for Android closes the gap

Posted:
in iPhone edited January 2014
Apple has improved its voice-driven personal assistant, Siri, with the iOS 7 update for iPhone and iPad, though Siri's chief competition, Google Now, has improved enough to be on par, new tests have concluded.

Siri


Analyst Gene Munster and his team at Piper Jaffray pitted Siri on iOS 7 against Android's Google Now in a barrage of new tests, and found that the two competing personal assistants are about even in terms of performance. Both were given a grade of "C+."

That was a marginal improvement for Siri, which earned a grade of "C" last year in iOS 6. With a check of 800 queries, Munster found that iOS 7 has improved Siri's ability to translate speech better in a noisy environment, with accuracy increasing from 83 percent to 94 percent.

The tests also found that Siri in iOS 7 has an improved ability to answer questions, with accuracy improving slightly from 77 percent to 79 percent.

Siri


iOS 7 has also further reduced Siri's reliance on Google for delivering data to users. While last December, Siri relied on Google for 27 percent of its data, according to tests by Piper Jaffray, Google now provides just 4 percent of Siri data, having been replaced largely by Microsoft Bing, Wolfram Alpha, and Wikipedia.

As for Google Now, its grade improved more significantly, from "D-" to "C+," with major improvements in comprehension and accuracy. In particular, Google Now's ability to understand users in a noisy environment grew from 78 percent accuracy in December 2012 to 92 percent in December 2013.

The tests show that Siri is still slightly more accurate than Google Now when both services correctly understand the user's query. While Siri has an 83 percent accuracy rate, according to Piper Jaffray, Google Now provided the correct answer 81 percent of the time.

Google Now


Still, Google Now's ability to correctly answer a question saw significant gains in the last year, growing from 72 percent accuracy to 81 percent. In the same period, Apple's Siri went from 81 percent accuracy to 83 percent.

In terms of total effectiveness, both Google Now and Siri are in a dead heat, with 79 percent accuracy rate in understanding queries presented by Piper Jaffray tests. As a result, both platforms were graded as a "C+" as of December 2013.
«134

Comments

  • Reply 1 of 72
    What about speed? Google Now is so much faster.
  • Reply 2 of 72
    Yes, but only Google Now will tell you how to talk to a girl, and then tell advertisers that you are dateless and have low self-esteem.
  • Reply 3 of 72
    Google now and Google voice search are different things. Munster proves what an idiot he is once again.
  • Reply 4 of 72
    danoxdanox Posts: 2,757member
    Actually Google Now is slower and is not the same thing as Siri.
  • Reply 5 of 72

    Piper Jaffray's report cards are pretty generous.  I think an outcome like "correctly completed request" would shed more light on the actual experience of interacting with SIRI.  As a novelty, it can be fun to play with, but in terms of clutch performance - like responding to a text on the go - it gets an F.

  • Reply 6 of 72
    peterbob wrote: »
    Google now and Google voice search are different things.

    That's what I was going to post. Google Now is the preemptive data service.
  • Reply 7 of 72
    nagrommenagromme Posts: 2,834member
    Siri vs. Now is one thing.

    But I also use dictation often to do searches, and Apple gets my words right most of the time, while Google almost never does. When Apple gets something wrong, I speak more carefully the second time and it works. With Google (searching in the YouTube app comes to mind) I can try, try again and just never get anything close to the words I spoke. In addition, Google responds much more slowly than Apple does. (I'm a native English speaker with no unusual accent.)

    So either Google's not all they're cracked up to be or (unlikely) they intentionally make their voice tech fail in their iOS apps. I haven't tested the Android versions.

    But two heads are better than one! The most common use I have for Siri: hold phone to my eat and say "Google search ___________." Great time saver!
  • Reply 8 of 72
    As for me... I would like Siri to do simple phone task oriented items like - "Play Bach Brandenberg Concerto" etc be performed local and not require a server. Of has that changed recently?
  • Reply 9 of 72
    danox wrote: »
    Actually Google Now is slower and is not the same thing as Siri.

    That's not my experience at all. Siri's achilles heal is its speed. It's just not noticeably slower than Google Now but slow enough to be annoyance to use.

    Also, when it can't connect and tells you try back later the message way too verbose since one is likely to want to repeat the request immediately.

    Finally, if you disable Siri it will default to the old voice commands which are all local. This should be what happens when it detects it can't connect instead of telling me it can't do something as basic and dialing a contact or adjusting a setting.
  • Reply 10 of 72
    danox wrote: »
    Actually Google Now is slower and is not the same thing as Siri.

    Check again.

    [VIDEO]
  • Reply 11 of 72
    rogifanrogifan Posts: 10,669member
    Piper Jaffray's report cards are pretty generous.  I think an outcome like "correctly completed request" would shed more light on the actual experience of interacting with SIRI.  As a novelty, it can be fun to play with, but in terms of clutch performance - like responding to a text on the go - it gets an F.
    I don't have that problem. Siri is much better for me since iOS 7 came out.
  • Reply 12 of 72
    solipsismxsolipsismx Posts: 19,566member
    dasanman69 wrote: »
    Actually Google Now is slower and is not the same thing as Siri.

    Check again.

    video:

    Siri looks like it wins in the first round, pretty much tie in the second and clearly loses the 3rd, although all three are fast enough. The problem with the 3rd round for Siri is that it doesn't list a map, but this is handy when you're driving as it can tell you how far it is away and if that sounds good; providing you preface your question with "Find me directions to..".

    The fourth test is useless because she states Siri had two phone numbers for the contact but doesn't say Google Now had two numbers. If it did, why did it assume they wanted that number? Test 6 with setting up an appointment Siri is fast but she has two emails for the contract which she then has to choose which also makes that test useless without additional information. If Google Now contacts lets you setup a primary so that it won't ask you which number or email or address to use unless you specify otherwise then it's a better system, but this doesn't indicate either way.

    On test 7,on setting up the reminder for milk, has the opposite issue where Google Now wants you to set which place to save the reminder. Can this setup so it saves it one place? If so, then it's a wash as both are fast, if not, then Siri wins.

    On the 8th test Google Now wins on playing the song but, again, I want to know if the device was pulling from iTunes Match which then means polling the server to stream the music or if it was local. I assume it was local but I also did a anecdotal test where my local music played faster than in the video but slower than in the video when it was pulling from iTunes server via iTM.

    Chrome definitely opened faster on Siri, which is surprising, but I find iOS's need to show you the Home Screen janky compared to the visuals provided by Google Now as it opened the app even though Now was slower.

    On the 10th test they both set up the email quickly and without issue but Google Now edged out Siri in speed, but not enough that Siri would feel slow.

    On the 11th test, setting up a timed reminder, they both were fast, but again Now edged out Siri but not to any great degree like we saw when Now was first introduced. What I don't get is why this reminder in Now doesn't have an option to where it gets saved to.

    With test 4 I am not sure who wins. I like the map overview before it zooms in on Apple Maps but I'm not sure if that's a feature or a way of hiding that it doesn't have the micro map location downloaded; or perhaps Google Maps doesn't have the macro location loaded. There is also the issue with each map telling you to go a different way? Which way is more correct?

    (stopped watching the video)

    In conclusion, that showed me that Siri, at least in this series of tests, has become much faster or that Google Now has become slower.


    PS: I really hate that when I use Apple Maps to say "Get me directions to a gas station along route." it will often find me ones in the opposite direction of travel. I'm usually on the highway when I use this feature and I don't want to turn backtrack to get gas. I'd expect that at the very least if I already have a destination set it should look for something ahead of me, but even if I didn't have a destination in maps it should know my direction of travel when in motion.
  • Reply 13 of 72

    Can the two services preform the same tasks? It's one thing that they translate and understand what you're asking, and other thing if the service cannot perform the task. I'm thinking specifically that Siri can buy your movie tickets for you and you can ask it something along the lines of "Can you remind me to pick up milk when I get to the grocery store?". Having no experience with Google's service I don't know whether it can do that or not

  • Reply 14 of 72
    solipsismxsolipsismx Posts: 19,566member
    Can the two services preform the same tasks? It's one thing that they translate and understand what you're asking, and other thing if the service cannot perform the task. I'm thinking specifically that Siri can buy your movie tickets for you and you can ask it something along the lines of "Can you remind me to pick up milk when I get to the grocery store?". Having no experience with Google's service I don't know whether it can do that or not

    Watch the video dasanman69 provided. I'd say it's just as robust as Siri although not sure about being able to buy movie tickets.

    I find Siri does have one major shortcoming compared to Google Now in terms to utility. If it's something that requires a browser you have to then access the Safari browser proper to read the page which means that you likely have to then unlock your device. With Google Now it will display the web contents of the search with considerably less rigamarole.
  • Reply 15 of 72
    dasanman69dasanman69 Posts: 13,002member
    solipsismx wrote: »
    Siri looks like it wins in the first round, pretty much tie in the second and clearly loses the 3rd, although all three are fast enough. The problem with the 3rd round for Siri is that it doesn't list a map, but this is handy when you're driving as it can tell you how far it is away and if that sounds good; providing you preface your question with "Find me directions to..".

    The fourth test is useless because she states Siri had two phone numbers for the contact but doesn't say Google Now had two numbers. If it did, why did it assume they wanted that number? Test 6 with setting up an appointment Siri is fast but she has two emails for the contract which she then has to choose which also makes that test useless without additional information. If Google Now contacts lets you setup a primary so that it won't ask you which number or email or address to use unless you specify otherwise then it's a better system, but this doesn't indicate either way.

    On test 7,on setting up the reminder for milk, has the opposite issue where Google Now wants you to set which place to save the reminder. Can this setup so it saves it one place? If so, then it's a wash as both are fast, if not, then Siri wins.

    On the 8th test Google Now wins on playing the song but, again, I want to know if the device was pulling from iTunes Match which then means polling the server to stream the music or if it was local. I assume it was local but I also did a anecdotal test where my local music played faster than in the video but slower than in the video when it was pulling from iTunes server via iTM.

    Chrome definitely opened faster on Siri, which is surprising, but I find iOS's need to show you the Home Screen janky compared to the visuals provided by Google Now as it opened the app even though Now was slower.

    On the 10th test they both set up the email quickly and without issue but Google Now edged out Siri in speed, but not enough that Siri would feel slow.

    On the 11th test, setting up a timed reminder, they both were fast, but again Now edged out Siri but not to any great degree like we saw when Now was first introduced. What I don't get is why this reminder in Now doesn't have an option to where it gets saved to.

    With test 4 I am not sure who wins. I like the map overview before it zooms in on Apple Maps but I'm not sure if that's a feature or a way of hiding that it doesn't have the micro map location downloaded; or perhaps Google Maps doesn't have the macro location loaded. There is also the issue with each map telling you to go a different way? Which way is more correct?

    (stopped watching the video)

    In conclusion, that showed me that Siri, at least in this series of tests, has become much faster or that Google Now has become slower.


    PS: I really hate that when I use Apple Maps to say "Get me directions to a gas station along route." it will often find me ones in the opposite direction of travel. I'm usually on the highway when I use this feature and I don't want to turn backtrack to get gas. I'd expect that at the very least if I already have a destination set it should look for something ahead of me, but even if I didn't have a destination in maps it should know my direction of travel when in motion.

    The problem with the test is that it's done on 2 different devices. I'd like to see a test done on 2 same model iPhones. Regardless of which one is 'better' they're both pretty darn good and offer us a feature that we only dreamt about a few short years ago.
  • Reply 16 of 72
    Quote:

    Originally Posted by SolipsismX View Post





    Watch the video dasanman69 provided. I'd say it's just as robust as Siri although not sure about being able to buy movie tickets.



    I find Siri does have one major shortcoming compared to Google Now in terms to utility. If it's something that requires a browser you have to then access the Safari browser proper to read the page which means that you likely have to then unlock your device. With Google Now it will display the web contents of the search with considerably less rigamarole.



    Thanks. I'll watch it when I get home

  • Reply 17 of 72
    dasanman69dasanman69 Posts: 13,002member

    Thanks. I'll watch it when I get home

    And the chick doing it is hot. :lol:
  • Reply 18 of 72

    1. Google was quick to see an important Apple innovation and move in that direction. Have they achieved parity? Don't know, since I've never used Google Now. But once again, Google is having to follow where Apple is leading. The same will be true of the fingerprint sensor and the 64-bit microprocessor.

    2. I am curious about what version of Android is required for Google Now to work, and what devices it's limited to. My guess is that there are far more Siri-capable devices in the wild than Google Now-capable devices, and that a larger percentage of capable Apple devices, versus Android devices, are used actively.

  • Reply 19 of 72
    dasanman69dasanman69 Posts: 13,002member
    sacto joe wrote: »
    <p style="border:0px;color:rgb(63,69,73);margin-bottom:15px;">1. Google was quick to see an important Apple innovation and move in that direction. Have they achieved parity? Don't know, since I've never used Google Now. But once again, Google is having to follow where Apple is leading. The same will be true of the fingerprint sensor and the 64-bit microprocessor.</p>

    <p style="border:0px;color:rgb(63,69,73);">2. I am curious about what version of Android is required for Google Now to work, and what devices it's limited to. My guess is that there are far more Siri-capable devices in the wild than Google Now-capable devices, and that a larger percentage of capable Apple devices, versus Android devices, are used actively.</p>

    Google Now requires 4.1+ and Siri was first available on the iPhone 4S. Last numbers I could find with a quick search was that Jellybean (4.1-4.3) was on roughly 50% of Android phones, so I'd say that you are correct.
  • Reply 20 of 72
    solipsismxsolipsismx Posts: 19,566member
    sacto joe wrote: »
    <p style="border:0px;color:rgb(63,69,73);margin-bottom:15px;">1. Google was quick to see an important Apple innovation and move in that direction. Have they achieved parity? Don't know, since I've never used Google Now. But once again, Google is having to follow where Apple is leading. The same will be true of the fingerprint sensor and the 64-bit microprocessor.</p>

    It's funny, every part of Siri was already available to Google from their previous experiments with GOOG-411 and other search endeavors but they did nothing to homogenize it into a viable, modern product until after Apple released Siri. It's also why they were able to make it so good so quickly, but how many of these monkey-see-monkey-do scenarios are available to them?

    Are they designing their own chipset now? The Nexus 5 is the phone I'd buy if I had to choose an Android-based device (although a big part of that is because of the OS update cycle is better and crapware is less) but I see no evidence of them moving to chip design at this point.
Sign In or Register to comment.