Siri accuracy dramatically improves in latest smart speaker showdown
Siri on HomePod improved its standing in an annual smart speaker comparison carried out by analyst Gene Munster's Loup Ventures, with Apple's virtual assistant weathering a barrage of accuracy tests to place second behind Google Assistant.
HomePod
For the test, Muster and fellow analyst Will Thompson asked four smart speakers -- Amazon Echo with Alexa, Google Home with Google Assistant, HomePod with Siri and and Harmon Kardon Invoke with Microsoft's Cortana -- a series of 800 questions and recorded the accuracy of the response, as well as the understanding of the query.
Munster and Thompson designed the test based on what most users would expect from a smart speaker. Questions fell into five categories: local points of interest, commerce, information, navigation and hardware commands.
Google assistant was able to understand all queries, of which it provided correct answers for about 88 percent. Siri came in second, understanding 99.6 percent of the queries and answering 74.6 percent correctly. That performance just ekes out Alexa, which understood 99.0 percent and correctly answered 72.5 percent. Cortana pulled up the rear only answering 63 percent of the questions correctly.
Siri excelled in the command category, which included many music related queries, of which HomePod specializes in thanks to native integration with Apple Music. This was the only one of the five categories where Siri came out on top.
Commerce was the area in which Siri was least knowledgeable, understandable as HomePod doesn't really allow you to buy anything as Alexa does. Alexa's eagerness to sell consumers products from Amazon stuff hampered its own score, however, as can be readily seen by asking "how much does a manicure cost."
Alexa recommends a manicure set for $60 dollars on Amazon, whereas Google Assistant gives an impressively detailed response, saying, "on average, a basic manicure will cost you about $20. However, special types of manicures like acrylic, gel, shellac, and no-chip range from about $20 to $50 in price, depending on the salon." Google's massive search engine capabilities clearly come into play here.
In terms of accuracy, Siri has improved 22 percentage points over the last nine months, where Google Assistant, Amazon Alexa, and Microsoft Cortana only improved between 7 and 9 points over the past year. As much flack as Apple's assistant has taken from critics of the service, these figures are quite telling.
The results shouldn't be entirely surprising given the relative investments each tech firm puts into their respective AI system, and it is encouraging to see Siri improve so drastically since the last time the test was run. When Loup Venture last conducted the evaluation in February, Siri came in dead last.
HomePod
For the test, Muster and fellow analyst Will Thompson asked four smart speakers -- Amazon Echo with Alexa, Google Home with Google Assistant, HomePod with Siri and and Harmon Kardon Invoke with Microsoft's Cortana -- a series of 800 questions and recorded the accuracy of the response, as well as the understanding of the query.
Munster and Thompson designed the test based on what most users would expect from a smart speaker. Questions fell into five categories: local points of interest, commerce, information, navigation and hardware commands.
Google assistant was able to understand all queries, of which it provided correct answers for about 88 percent. Siri came in second, understanding 99.6 percent of the queries and answering 74.6 percent correctly. That performance just ekes out Alexa, which understood 99.0 percent and correctly answered 72.5 percent. Cortana pulled up the rear only answering 63 percent of the questions correctly.
Smart speaker test results | Source: Loup Ventures
Siri excelled in the command category, which included many music related queries, of which HomePod specializes in thanks to native integration with Apple Music. This was the only one of the five categories where Siri came out on top.
Commerce was the area in which Siri was least knowledgeable, understandable as HomePod doesn't really allow you to buy anything as Alexa does. Alexa's eagerness to sell consumers products from Amazon stuff hampered its own score, however, as can be readily seen by asking "how much does a manicure cost."
Alexa recommends a manicure set for $60 dollars on Amazon, whereas Google Assistant gives an impressively detailed response, saying, "on average, a basic manicure will cost you about $20. However, special types of manicures like acrylic, gel, shellac, and no-chip range from about $20 to $50 in price, depending on the salon." Google's massive search engine capabilities clearly come into play here.
In terms of accuracy, Siri has improved 22 percentage points over the last nine months, where Google Assistant, Amazon Alexa, and Microsoft Cortana only improved between 7 and 9 points over the past year. As much flack as Apple's assistant has taken from critics of the service, these figures are quite telling.
The results shouldn't be entirely surprising given the relative investments each tech firm puts into their respective AI system, and it is encouraging to see Siri improve so drastically since the last time the test was run. When Loup Venture last conducted the evaluation in February, Siri came in dead last.
Comments
The example they give is “Can you order me more paper towels”. Wouldn’t that be a “no” for HomePod? But HomePod’s score for commerce is 56% vs 52% for Echo. What?
This doesn’t make sense to me. Can anything be purchased via HomePod?
Edit: I read further, turns out the questions weren’t necessarily to buy something but to learn about products (where to buy, how much, etc.).
In the end, the fact remains that Siri gets it wrong 25% of the time and Apple still has a lot of work to do.
For example, I make a point of ‘loving’ specific songs on my iPhone/Mac. You’d think that asking Siri to play my loved tracks from the latest Disturbed album would therefore be an obvious easy win. Nope, Siri just plays the latest album.
Ask it to play original artist only tracks from Queen and you’ll get a Queen mix that has some songs that have Queen dubbed in with other artists too.
Or maybe it's already possible. I mostly use Google home, have Alexa but don't use it and stopped using Siri
It is possible... for techies. Which is a problem,
I mean thats kind of what they all do, but the other assistants read the website. Or a summary of it.
EDIT: Nope that won't work either.
A (tightwad) friend of mine recently splurged and bought a $20 Echo Dot. Said he could not believe how well it worked. Followed up with SIRI sucks. Siri on my iPhone 8+ doesn't seem to understand what I request. I'm not going to buy a HomePod for the same experience even if Apple is desperate enough to sell the HomePod for $250.
Wake me up when queries like this (with these imperfect, human phrasing) work as we'd like:
"If I need to get to Brian's house by 6:00 what time should I leave the house? Oh yeah, I need to stop for gas along the way. Let me know 5 minutes before I should leave."
Just for giggles I tried this on Google Home, but asking "If I need to be in downtown Tampa by 6pm tonight what time should I leave". and followed up in the same session with "remind me to leave at (x):00. Worked fine. The additional time needed to get gas too would be impossible to answer of course. Things are gradually becoming a little easier. Baby steps.
Apple is always held by an impossible standard so unless it's 5x better, Siri is a "Fail".
Siri was never as bad as the media would like you to believe.
Apple needs to acquire Hue or a few companies to really hit home hard.