Test finds HomePod's Siri 'at the bottom of the totem pole' in smartspeaker AI

Posted:
in General Discussion
While the Apple HomePod is the "best sounding" smartspeaker and has a "measurably better" user experience in many areas, its underlying AI assistant -- Siri -- failed dramatically in a query test versus Amazon Alexa, Google Assistant, and Microsoft Cortana, according to Loup Ventures.




Three HomePods were subjected to 782 queries by the firm, said analyst Gene Munster. While Siri understood 99.4 percent of them, it was only able to answer 52.3 percent of them correctly. The latter figure compares with rates of 81 percent for Google, 64 percent for Alexa, and 57 percent for Cortana.

Siri did beat out Alexa and Cortana in test categories like "Local" (e.g. "Where can I find a good cup of coffee around here?") and "Commerce" ("Help me buy some new shoes"), but still failed in part because a number of functions -- such as email, calling, calendars, and navigation -- are simply inaccessible on a HomePod. Likewise, where Siri would display a list of Google search results on an iPhone, it outright fails to provide anything on a HomePod.

Removing navigation, calling, email, and calendar-related categories improves Siri's correct answer rate to 67 percent, and Munster argues that Apple will probably improve the HomePod's options over time, making it "vastly more useful and integrated with your other Apple devices."

The Loup Ventures analysis lines up with early HomePod reviews, which have typically pegged Siri as the weakest link. Apple has directed its marketing focus to sound quality, emphasizing technologies like beamforming and the use of seven separate tweeters.

Loup Ventures is forecasting sales of 7 million HomePods in 2018, boosting Apple's revenue and earnings by 1 percent. Sales are expected to hit 10.9 million in 2019, and grow 40 to 45 percent annually during the next three years.
«13456

Comments

  • Reply 1 of 113
    Disclaimer: I didn’t read the article. I’m just commenting on the title. 

    I would personally much rather be taking things slow with all of this new artificial intelligence than blindly following the race to the first without ever thinking about the potential consequences. 

    That aside... whatever we we may say about who is first and who is best. Majority of people still don’t even talk to their assistants. 
    rob53tmaymagman1979jonagoldRobPalmer9propodrandominternetpersonbshankadamclolliver
  • Reply 2 of 113
    jungmarkjungmark Posts: 6,926member
    Considering Apple is marketing it as a high end speaker, it’s not surprising Siri is lacking. But it’ll improve with updates. You can’t improve hardware once the device is sold. 
    magman1979RobPalmer9trashman69chasmbluefire1bshanklollivertechprod1gyjony0
  • Reply 3 of 113
    I hope more people doing a test like this. If reports like this won’t push Apple to be serious about SIRI I don’t know what will. 

    There’re so many fundamental things SIRI could be improved. If Apple won’t do it I hope they get called out like this until they understand it will hurt their brand. 

    (I’m talking about SIRI in general, like on iPhone, not specifically on HomePod.)
    edited February 2018 superklotonphil sukalewskipatchythepiratekitatitwaverboyGeorgeBMacscafe2viclauyycjony0
  • Reply 4 of 113
    jungmark said:
    Considering Apple is marketing it as a high end speaker, it’s not surprising Siri is lacking. But it’ll improve with updates. You can’t improve hardware once the device is sold. 
    There are general questions Siri can answer on the phone but not HomePod. That makes no sense to me. An example: which country has won the most gold medals at the 2018 Winter Olympics. Siri on iPhone answers the question. On HomePod Siri says I can’t answer that question. Why? Why all these different versions of Siri?
    supadav03superklotonking editor the gratepatchythepirateGeorgeBMacavon b7airnerdjony0
  • Reply 5 of 113
    evilutionevilution Posts: 1,399member
    HomePod is a quality speaker with a poor voice assistant added on. It’s sold as a speaker.
    The Echo is a poor speaker with a quality voice assistant added on. It’s sold as a voice assistant.

    However, software updates won’t improve the sound quality of the Echo but a software update can improve Siri.

    Hopefully all of the reviews will encourage Apple to improve Siri.
    Then hopefully that improvement will force Amazon to improve Alexa. Ideally to understand more than 3 languages.

    That being said, who really ever uses a voice assistant? I set alarms for cooking and that’s it.

    edited February 2018 superklotontmaymagman1979RobPalmer9brian greenStrangeDaysjony0
  • Reply 6 of 113
    linkmanlinkman Posts: 1,035member
    I can believe that Siri scored lowest on the list. Apple acquired it in 2010 and it hasn't improved or expanded in capabilities much since its intro with the 4S in 2011 (do they still consider it in beta status?). Apple Maps seems to be lagging behind the competition too -- I've seen some insane suggestions that are literally over 1000 miles away favored over a logical local location. 
    patchythepiratebrian greenkitatitjony0
  • Reply 7 of 113
    evilution said:
    HomePod is a quality speaker with a poor voice assistant added on. It’s sold as a speaker.
    The Echo is a poor speaker with a quality voice assistant added on. It’s sold as a voice assistant.

    However, software updates won’t improve the sound quality of the Echo but a software update can improve Siri.

    Hopefully all of the reviews will encourage Apple to improve Siri.
    Then hopefully that improvement will force Amazon to improve Alexa. Ideally to understand more than 3 languages.

    That being said, who really ever uses a voice assistant? I set alarms for cooking and that’s it.

    And you can only set one timer with Siri and you can’t name it. Apple should be the leader in this space.
    patchythepiratebrian greenmuthuk_vanalingam
  • Reply 8 of 113
    I bought 2 HomePods, different rooms.
    As a Speaker version of an iPod, it's outstanding  - I can stream music from the BBC over airplay, or BBC talk, and it's so clear its uncanny. the frequency response seems incredibly neutral - some pieces of music sound as good as when using pretty decent headphones.

    HomePod also doubles as a really very good speakerphone - I called my son using FaceTime audio on his iPhone X, and the call quality was very very good, it was as though he was in the same room as myself.

    Siri on HomePod will get better I assume, but I don't really understand why its so poor out of the gate - no multiple timers, inability to answer simple questions that Siri on my iPhone knows, I could go on -  Per a comment above, I for one often as Siri (or Alexa - I got a dot as a Christmas gift) for asking a simple question.
    brian green
  • Reply 9 of 113
    linkman said:
    I can believe that Siri scored lowest on the list. Apple acquired it in 2010 and it hasn't improved or expanded in capabilities much since its intro with the 4S in 2011 (do they still consider it in beta status?). Apple Maps seems to be lagging behind the competition too -- I've seen some insane suggestions that are literally over 1000 miles away favored over a logical local location. 
    You must not have read it carefully.  It's not that Apple hasn't improved Siri,  it's just that in the initial roll out of Homepod,  Apple hasn't activated all the domains.  If you take that into account,  the Homepod would be more able than Echo or Cortana and much closer to Google.
    tmaybshankNY1822lolliver
  • Reply 10 of 113
    matrix077 said:
    I hope more people doing a test like this. If reports like this won’t push Apple to be serious about SIRI I don’t know what will. 

    There’re so many fundamental things SIRI could be improved. If Apple won’t do it I hope they get called out like this until they understand it will hurt their brand. 

    (I’m talking about SIRI in general, like on iPhone, not specifically on HomePod.)
    Totally agreed. Siri is useless for sending a text and with Maps. It's maddening in downtown Atlanta (on AT&T) that my text gets noted correctly when I speak until it goes to Apple's servers and Siri converts it to nonsense. 

    Apple is so focused with gimmick tech for the tweenies that there stuff no longer "just works".
    hammeroftruthbrian greenanantksundaramkitatittokyojimukrawallwaverboy
  • Reply 11 of 113
    gatorguygatorguy Posts: 24,176member
    evilution said:
    That being said, who really ever uses a voice assistant? I set alarms for cooking and that’s it.

    I had to use it a little earlier to find my darn phone. "Hey Google find my phone" will ring only mine at full volume. Not having to run around the house room by room looking for it or asking my wife to call it ( and hoping the ringer is turned up) is almost by itself worth the price of admission.  :)
    edited February 2018 brian greenlorin schultzviclauyyccropr
  • Reply 12 of 113
    genovellegenovelle Posts: 1,480member
    gatorguy said:
    evilution said:
    That being said, who really ever uses a voice assistant? I set alarms for cooking and that’s it.

    I had to use it a little earlier to find my darn phone. "Hey Google find my phone" will ring only mine at full volume. Not having to run around the house room by room looking for it or asking my wife to call it ( and hoping the ringer is turned up) is almost by itself worth the price of admission.  :)
    So it knows your voice out of the multitude of voices that could be talking to it. That is impressive. I don’t know why Google is not advertising that ability. 

    brian green
  • Reply 13 of 113
    genovellegenovelle Posts: 1,480member
    matrix077 said:
    I hope more people doing a test like this. If reports like this won’t push Apple to be serious about SIRI I don’t know what will. 

    There’re so many fundamental things SIRI could be improved. If Apple won’t do it I hope they get called out like this until they understand it will hurt their brand. 

    (I’m talking about SIRI in general, like on iPhone, not specifically on HomePod.)
    Totally agreed. Siri is useless for sending a text and with Maps. It's maddening in downtown Atlanta (on AT&T) that my text gets noted correctly when I speak until it goes to Apple's servers and Siri converts it to nonsense. 

    Apple is so focused with gimmick tech for the tweenies that there stuff no longer "just works".
    Not sure what you are talking about I use Siri everyday and used it for over a year doing deliverys and it worked flawlessly.

    edited February 2018 lolliver
  • Reply 14 of 113
    gatorguygatorguy Posts: 24,176member
    genovelle said:
    gatorguy said:
    evilution said:
    That being said, who really ever uses a voice assistant? I set alarms for cooking and that’s it.

    I had to use it a little earlier to find my darn phone. "Hey Google find my phone" will ring only mine at full volume. Not having to run around the house room by room looking for it or asking my wife to call it ( and hoping the ringer is turned up) is almost by itself worth the price of admission.  :)
    So it knows your voice out of the multitude of voices that could be talking to it. That is impressive. I don’t know why Google is not advertising that ability. 

    They do advertise it. They're just darn lousy at promotion. Confounding considering they earn substantial income from ad placement.
    http://www.androidpolice.com/2018/02/10/ultimate-guide-google-home-tips-tricks-understanding-making-assistant-speaker/
    edited February 2018 muthuk_vanalingam
  • Reply 15 of 113
    davendaven Posts: 696member
    jungmark said:
    Considering Apple is marketing it as a high end speaker, it’s not surprising Siri is lacking. But it’ll improve with updates. You can’t improve hardware once the device is sold. 
    There are general questions Siri can answer on the phone but not HomePod. That makes no sense to me. An example: which country has won the most gold medals at the 2018 Winter Olympics. Siri on iPhone answers the question. On HomePod Siri says I can’t answer that question. Why? Why all these different versions of Siri?
    Decided to ask your gold medal question to Siri on my iWatch and it answered no problem. Odd that Siri on HomePod can't answer it. Regardless, I'm not considering the HomePod for Siri capabilities but as a speaker for the TV in my bedroom where I don't want it telling me what to do. ;-)
    lolliver
  • Reply 16 of 113
    This brings up something I’ve had a problem with regarding asking Siri a question, and getting back, “Here’s what I found on the web.”  And then I go into an expletive laced tirade.  I didn’t ask for you to pull up a google page, nor did I ask you to find me some stupid list.  I asked for, and expect, an actual answer! Why bother showing me a website and all when I could have Googled that myself?  If I ask a question, I want a verbal answer back.  Until that happens every single time, I can’t use Siri because I get too annoyed as BS websites as answers.  Fix it!
    anantksundaramravnorodommuthuk_vanalingam
  • Reply 17 of 113
    Siri sucks, period.

    Get rid of this embarrassment, Apple; Man up, call it quits, buy IBM’s Watson, and build on it. Siri-ously.
    brian greenchristophbporg1969muthuk_vanalingam
  • Reply 18 of 113

    evilution said:
    HomePod is a quality speaker with a poor voice assistant added on. It’s sold as a speaker.
    The Echo is a poor speaker with a quality voice assistant added on. It’s sold as a voice assistant.

    However, software updates won’t improve the sound quality of the Echo but a software update can improve Siri.

    Hopefully all of the reviews will encourage Apple to improve Siri.
    Then hopefully that improvement will force Amazon to improve Alexa. Ideally to understand more than 3 languages.

    That being said, who really ever uses a voice assistant? I set alarms for cooking and that’s it.

    Wow. I am sure your use case = The World’s.
    brian green
  • Reply 19 of 113
    Serious question: do people really ask their home-centric smart speakers for navigation? If so, how does that work and why is it better than just asking my phone?

    Thanks in advance. 
    EsquireCatsrandominternetpersonlolliverporg1969
  • Reply 20 of 113
    It is an amazing speaker! Come on guys, you are really complaining about SIRI's VR on what is an amazing sounding speaker? I have been listening to mine all day today, and it blows me away. I have been an audio connoisseur for 40 years. As many others have said, the acoustic mechanicals and hardware are not changeable. The SIRI software totally is. That is all there is to it. Those POS Google and Amazon products are lame in the sound department, and can never get better. But whenever Apple is ready to open up SIRI, it will be a minor software update. All of you complainers need to close your pie holes, really.
    randominternetpersonlolliverporg1969
Sign In or Register to comment.