Siri accuracy dramatically improves in latest smart speaker showdown

Posted:
in iOS edited December 2018
Siri on HomePod improved its standing in an annual smart speaker comparison carried out by analyst Gene Munster's Loup Ventures, with Apple's virtual assistant weathering a barrage of accuracy tests to place second behind Google Assistant.

HomePod
HomePod


For the test, Muster and fellow analyst Will Thompson asked four smart speakers -- Amazon Echo with Alexa, Google Home with Google Assistant, HomePod with Siri and and Harmon Kardon Invoke with Microsoft's Cortana -- a series of 800 questions and recorded the accuracy of the response, as well as the understanding of the query.

Munster and Thompson designed the test based on what most users would expect from a smart speaker. Questions fell into five categories: local points of interest, commerce, information, navigation and hardware commands.

Google assistant was able to understand all queries, of which it provided correct answers for about 88 percent. Siri came in second, understanding 99.6 percent of the queries and answering 74.6 percent correctly. That performance just ekes out Alexa, which understood 99.0 percent and correctly answered 72.5 percent. Cortana pulled up the rear only answering 63 percent of the questions correctly.

Smart Speaker Test ResultsSmart speaker test results | Source: Loup Ventures


Siri excelled in the command category, which included many music related queries, of which HomePod specializes in thanks to native integration with Apple Music. This was the only one of the five categories where Siri came out on top.

Commerce was the area in which Siri was least knowledgeable, understandable as HomePod doesn't really allow you to buy anything as Alexa does. Alexa's eagerness to sell consumers products from Amazon stuff hampered its own score, however, as can be readily seen by asking "how much does a manicure cost."

Alexa recommends a manicure set for $60 dollars on Amazon, whereas Google Assistant gives an impressively detailed response, saying, "on average, a basic manicure will cost you about $20. However, special types of manicures like acrylic, gel, shellac, and no-chip range from about $20 to $50 in price, depending on the salon." Google's massive search engine capabilities clearly come into play here.

In terms of accuracy, Siri has improved 22 percentage points over the last nine months, where Google Assistant, Amazon Alexa, and Microsoft Cortana only improved between 7 and 9 points over the past year. As much flack as Apple's assistant has taken from critics of the service, these figures are quite telling.

The results shouldn't be entirely surprising given the relative investments each tech firm puts into their respective AI system, and it is encouraging to see Siri improve so drastically since the last time the test was run. When Loup Venture last conducted the evaluation in February, Siri came in dead last.
«1

Comments

  • Reply 1 of 24
    I’d like to see more of the questions they asked. HomePod did much better under “Commerce” than I would expect.

    The example they give is “Can you order me more paper towels”. Wouldn’t that be a “no” for HomePod? But HomePod’s score for commerce is 56% vs 52% for Echo. What?

    This doesn’t make sense to me.  Can anything be purchased via HomePod?

    Edit: I read further, turns out the questions weren’t necessarily to buy something but to learn about products (where to buy, how much, etc.). 
    edited December 2018 watto_cobra
  • Reply 2 of 24
    Glad to see Apple is improving Siri. I’d love to see her be able to answer several questions in a row, rather than having to say “hey Siri” for each one. She could ask, “anything else?” like the drive through lol, before she stops listening. That would keep privacy. 
    chiawatto_cobragilly33
  • Reply 3 of 24
    MplsPMplsP Posts: 1,105member
    Glad to see the improvements - my experience with Siri on my phone is decidedly mixed. Asking her to play a song goes well, but asking her an actual question will most often get an answer like ‘I found this on the web...’ essentially Siri defaults to doing a google search; not terribly helpful. If all I wanted was a google search, I could use safari or the google app. 
    In the end, the fact remains that Siri gets it wrong 25% of the time and Apple still has a lot of work to do. 
    chiawatto_cobra
  • Reply 4 of 24
    saareksaarek Posts: 1,105member
    I love my HomePod, but Siri misses some Apple Music specific features that you’d really think would just be in there.

    For example, I make a point of ‘loving’ specific songs on my iPhone/Mac. You’d think that asking Siri to play my loved tracks from the latest Disturbed album would therefore be an obvious easy win. Nope, Siri just plays the latest album.

    Ask it to play original artist only tracks from Queen and you’ll get a Queen mix that has some songs that have Queen dubbed in with other artists too.
    watto_cobra
  • Reply 5 of 24
    I want to see these assistants understanding fairly basic home automation tasks better. I could go into my Hue or Nest app and set up a timer or a schedule but i want to be able to ask my assistant simply "Turn on the heating now and turn it off in 30 minutes" or "turn on the front light at 10pm for an hour" the more gear i get, the more i need a central management system that uses voice for ad hoc requests

    Or maybe it's already possible. I mostly use Google home, have Alexa but don't use it and stopped using Siri
    edited December 2018 williamlondon
  • Reply 6 of 24
    asdasdasdasd Posts: 5,233member
    great, but now Apple needs to get into the home assistant market. Although it may be too late

    williamlondon
  • Reply 7 of 24
    asdasdasdasd Posts: 5,233member

    I want to see these assistants understanding fairly basic home automation tasks better. I could go into my Hue or Nest app and set up a timer or a schedule but i want to be able to ask my assistant simply "Turn on the heating now and turn it off in 30 minutes" or "turn on the front light at 10pm for an hour" the more gear i get, the more i need a central management system that uses voice for ad hoc requests

    Or maybe it's already possible. I mostly use Google home, have Alexa but don't use it and stopped using Siri
    It is possible... for techies. Which is a problem,
  • Reply 8 of 24
    asdasdasdasd Posts: 5,233member

    MplsP said:
    Glad to see the improvements - my experience with Siri on my phone is decidedly mixed. Asking her to play a song goes well, but asking her an actual question will most often get an answer like ‘I found this on the web...’ essentially Siri defaults to doing a google search; not terribly helpful. If all I wanted was a google search, I could use safari or the google app. 
    In the end, the fact remains that Siri gets it wrong 25% of the time and Apple still has a lot of work to do. 
    I mean thats kind of what they all do, but the other assistants read the website. Or a summary of it. 
  • Reply 9 of 24
    In some ways it doesn't matter how much Siri improves, people will still say Siri sucks. The negative perception is out there and it will take a long time to get rid of that perception. It's just like with Apple Maps....people still say it sucks really bad as if Apple hasn't done anything with it since its conception. 
    williamlondonmike1bonobobasdasdStrangeDayswatto_cobra
  • Reply 10 of 24
    gatorguygatorguy Posts: 20,030member
    I want to see these assistants understanding fairly basic home automation tasks better. I could go into my Hue or Nest app and set up a timer or a schedule but i want to be able to ask my assistant simply "Turn on the heating now and turn it off in 30 minutes" or "turn on the front light at 10pm for an hour" the more gear i get, the more i need a central management system that uses voice for ad hoc requests

    Or maybe it's already possible. I mostly use Google home, have Alexa but don't use it and stopped using Siri
    Threading two or three commands together in one sentence fails often I agree. Heck Google Assistant only somewhat recently started responding to more than one request within a session, ex. Hey Google turn off the living room lights and set the downstairs thermostat to 65. Parsing a sentence like you wrote will be a long time coming I suspect. Even another human could stumble over what you want done.  Maybe if you reworded your request to something like "Hey Google set the thermostat to 70 and in an hour set the thermostat to 60"? I've not tried that type of combination myself and it probably won't work either but worth a try.

    EDIT: Nope that won't work either.
    edited December 2018 philboogie
  • Reply 11 of 24
    Anilu_777 said:
    Glad to see Apple is improving Siri. I’d love to see her be able to answer several questions in a row, rather than having to say “hey Siri” for each one. She could ask, “anything else?” like the drive through lol, before she stops listening. That would keep privacy. 
    I like the anything else idea 💡 

    watto_cobra
  • Reply 12 of 24
    It’s funny how much hate Apple gets for Siri when Google and Microsoft laughed at the whole concept of a digital assistant when it came out, say no one wants to talk to their phone and wouldn’t work. They continue to fail to have the vision to see value in something but then scrambled to copy it. People forget that Apple is a hardware company that makes software because they can’t trust partners to do so without ending up with a knife in the back. (Google & Microsoft) These two companies are software companies they Apple partnered with to do what they do best so they can focus on their strengths. These guys chose to use that partnership to prepare to compete directly with Apple. 
    StrangeDayswatto_cobraphilboogie
  • Reply 13 of 24
    peteopeteo Posts: 348member
    If only i could get my apple watch to even transcribe my question and not say hold on a minute, i'll tap you when I'm ready. then do nothing. This happens over 50% of the time now with my series 4 and its making siri useless.
    williamlondonphilboogie
  • Reply 14 of 24
    k2kwk2kw Posts: 1,647member
    macxpress said:
    In some ways it doesn't matter how much Siri improves, people will still say Siri sucks. The negative perception is out there and it will take a long time to get rid of that perception. It's just like with Apple Maps....people still say it sucks really bad as if Apple hasn't done anything with it since its conception. 
    Because SIRI needs to improve faster because it has such a bad reputation now.   Alexa seems to be always adding skills and they advertise them.   Many may seem stupid but it does give the impression of constant improvement and innovation.    Those 10,000 people working on Alexa are making a difference.     Maybe Apple should hold a mini Siri Software Development conference for 3 days.   They need to be releasing improvements to SIRI every month not waiting to announce 1 or 2 things at WWDC.

    A (tightwad) friend of mine recently splurged and bought a $20 Echo Dot.   Said he could not believe how well it worked. Followed up with SIRI sucks.   Siri on my iPhone 8+ doesn't seem to understand what I request.    I'm not going to buy a HomePod for the same experience even if Apple is desperate enough to sell the HomePod for $250.    
       
  • Reply 15 of 24
    MplsPMplsP Posts: 1,105member
    asdasd said:

    MplsP said:
    Glad to see the improvements - my experience with Siri on my phone is decidedly mixed. Asking her to play a song goes well, but asking her an actual question will most often get an answer like ‘I found this on the web...’ essentially Siri defaults to doing a google search; not terribly helpful. If all I wanted was a google search, I could use safari or the google app. 
    In the end, the fact remains that Siri gets it wrong 25% of the time and Apple still has a lot of work to do. 
    I mean thats kind of what they all do, but the other assistants read the website. Or a summary of it. 
    exactly - when you ask, they actually give you useful information rather than simply saying "go to this website and read for yourself." That's a critical difference that makes a big impact on how useful the answers actually are. The first is little more than text-to-speach filling in the search line for google; really not much AI involved there. 
  • Reply 16 of 24
    dewmedewme Posts: 1,943member
    I think the general public's perception of what these voice assistants should be able to do does not reflect the reality of what they currently are. There are some slivers of artificial intelligence (AI) and machine learning (ML) in play, mostly around voice recognition and associations, but for the most part these are simply voice based natural language query machines with a simple state machine behind the query responder. With the current state of this technology you should basically expect to get similar results to your voice queries that you would get from typing a query into a search engine. The fact that Google's devices perform better than anyone else's is because Google's query and search infrastructure is more mature and robust than that of the other companies. Amazon's Alexa is seen as more natural and personable because they've tweaked the state machine in the responder to follow up with prompts and suggestions, which are for the most part canned. Alexa also has a lot of dog & pony tricks like telling stupid jokes or saying goodnight in a friendly tone, but only when prompted to perform the trick.

    Also, as a user of these products you are also adapting your interaction patterns and queries to achieve better results and fit into the query model, patterns, and "vocabulary" that each products presents. Users who are willing to adapt to the device generally believe that the device is "smarter" that users who do not adapt to fit the device. Siri on the HomePod, for example, is very adept at topics relating to music and all of the metadata surrounding music, artists, and genres. But if you query your HomePod about general topics, say astronomy or geography, the HomePod is less likely to satisfy your query in a satisfying way. In essence these devices/systems currently have personalities and specialization of "knowledge" in the same way humans and companion animals in your life do. Asking your gearhead buddy about the dilemmas presented by dark energy is unlikely to yield a productive answer versus asking your astrophysicist sister the same. People have a finite set of knowledge to draw upon and so do the processing engines and associative information models behind Siri and Alexa. My experience with Siri on the HomePod is very satisfying because I interact with it as if it is a music savant, not a general purpose or all knowing source of information and understanding. Over time I expect more useful AI to get integrated into these products, but for now they are all very basic but still very helpful time saving products. Except when they aren't, but even then it's no big deal.

    randominternetperson
  • Reply 17 of 24
    I think we're at the place with voice assistants that the smart phone market was before the iPhone.  In a real sense, they all suck.  None of them work the way that we want (think Star Trek).  Basically all these companies are competing in a very public Turing test.  When you can talk to your device the way you can talk to a friend or a doctor and consistently get good, succinct answers, we'll have arrived.  Google demoed some things this year that were very impressive, but there's a big gap between demoware and shipping products (as Siri has shown).

    Wake me up when queries like this (with these imperfect, human phrasing) work as we'd like:

    "If I need to get to Brian's house by 6:00 what time should I leave the house? Oh yeah, I need to stop for gas along the way. Let me know 5 minutes before I should leave."


  • Reply 18 of 24
    gatorguygatorguy Posts: 20,030member
    I think we're at the place with voice assistants that the smart phone market was before the iPhone.  In a real sense, they all suck.  None of them work the way that we want (think Star Trek).  Basically all these companies are competing in a very public Turing test.  When you can talk to your device the way you can talk to a friend or a doctor and consistently get good, succinct answers, we'll have arrived.  Google demoed some things this year that were very impressive, but there's a big gap between demoware and shipping products (as Siri has shown).

    Wake me up when queries like this (with these imperfect, human phrasing) work as we'd like:

    "If I need to get to Brian's house by 6:00 what time should I leave the house? Oh yeah, I need to stop for gas along the way. Let me know 5 minutes before I should leave."


    Whoa...
    Just for giggles I tried this on Google Home, but asking "If I need to be in downtown Tampa by 6pm tonight what time should I leave". and followed up in the same session with "remind me to leave at (x):00.  Worked fine. The additional time needed to get gas too would be impossible to answer of course. Things are gradually becoming a little easier. Baby steps. 
  • Reply 19 of 24
    AppleExposedAppleExposed Posts: 584unconfirmed, member
    The problem is, Siri is the original Siri by APPLE.

    Apple is always held by an impossible standard so unless it's 5x better, Siri is a "Fail".

    Siri was never as bad as the media would like you to believe.
    watto_cobra
  • Reply 20 of 24
    AppleExposedAppleExposed Posts: 584unconfirmed, member
    asdasd said:
    great, but now Apple needs to get into the home assistant market. Although it may be too late

    Apple was first. They just drug their feet.

    Apple needs to acquire Hue or a few companies to really hit home hard.
Sign In or Register to comment.