Future HomePod may only answer Siri queries if you look at it

Posted:
in Future Apple Hardware edited December 2020
Apple device users may not necessarily have to state "Hey Siri" to invoke the digital assistant, with Apple looking into ways to use gaze detection to automatically trigger verbal control of a device, without needing a vocal prompt.

Apple HomePod


Owners of multiple devices in the Apple ecosystem will be familiar with one of the lesser-known problems of using Siri, namely getting it to work on one device and not another. When you are in a room that contains an iPhone, an iPad, and a HomePod mini, it can be hard to work out which device will actually respond to a query, and it may not necessarily be the desired device at that.

Furthermore, not everyone feels comfortable with the "Hey Siri" prompt being used at all. For example, errant uses of "Hey Siri" and trigger phrases of other digital assistants on television and radio can cause queries to be made that users may not want to take place.

There is also the possibility of users needing to interact with devices without using their voice at all. In situations where a command may need to be issued to hardware from a distance, it may not be possible to use the auditory control scheme, or another control mechanism.

In a patent granted by the US Patent and Trademark Office on Tuesday titled "Device control using gaze information," Apple suggests it may be possible to use a user's gaze to determine if they want assistance from Siri or another system, without requiring the initial verbal prompt.

The filing suggests a system which uses cameras and other sensors capable of determining the location of a user and the path of their gaze, to work out what they are looking at. This information could be used to automatically set the looked-at device to go into an instruction-accepting mode where it actively listens, in the expectation that instructions will be told to it.

If the digital assistant interprets what could be a command in this state, it can then carry it out as if the verbal trigger was said beforehand, saving users from a step. This would still allow for phrases like "Hey Siri" to function, especially when the user isn't looking at the device.

Using the gaze as a barometer for whether the user wants to tell the digital assistant a command is also useful in other ways. For example, gaze detected looking at the device could signal an intent by the user that the device should follow instructions.

A digital assistant for a HomePod could potentially only interpret a command if the user is looking at it, the patent suggests.
A digital assistant for a HomePod could potentially only interpret a command if the user is looking at it, the patent suggests.


In practical terms, this could mean the difference between the device interpreting a sentence fragment such as "play Elvis" as a command or as part of a conversation that it should otherwise ignore.

For owners of multiple devices, gaze detection could allow for an instruction to be made to one device and not others, singled out by looking at it.

The filing mentions that simply looking at the device won't necessarily register as an intention for it to listen for instruction, as a set of "activation criteria" needs to be met. This could merely consist of a continuous gaze for a period of time, like a second, to eliminate minor glances or false positives from a person turning their head.

The angle of the head position is also important. For example, if the device is located on a bedside cabinet and the user is laid in bed asleep, the device could potentially count the user facing the device as a gaze depending on how they lie, but could discount it as such for realizing the user's head is on its side instead of vertical.

In response, a device could provide a number of indicators to the user that the assistant has been activated by a glance, such as a noise or a light pattern from built-in LEDs or a display.

Given the ability to register a user's gaze, it would also be feasible for the system to detect if the user is looking at an object they want to interact with, rather than the device holding the virtual assistant. For example, a user could be looking at one of multiple lamps in the room, and the device could use the context of the user's gaze to work out which lamp the user wants turned on from a command.

Nearby devices could detect the user's gaze of other controllable objects in a room.
Nearby devices could detect the user's gaze of other controllable objects in a room.


Originally filed on August 28, 2019, the patent lists its inventors as Sean B. Kelly, Felipe Bacim De Araujo E Silva, and Karlin Y. Bark.

Apple files numerous patent applications on a weekly basis, but while the existence of a patent indicates areas of interest for its research and development efforts, they do not guarantee the patent will be used in a future product or service.

Being able to remotely operate a device has cropped up in earlier patent filings a few times. For example, a 2015 patent for a "Learning-based estimation of hand and finger pose" suggested the use of an optical 3D mapping system for hand gestures, one that could feasibly be similar in concept to Microsoft's Kinect hardware.

Another filing from 2018 for a "Multi media computing or entertainment system for responding to user presence and activity" proposes the introduction of three-dimensional sensor data to produce a depth map of the room, one which could be used for whole-room gesture recognition.

Gaze detection has also been explored. In March, Apple applied for a patent titled "Gaze-dependent display encryption," one which measured where the user was looking on the screen. In areas of a user's active vision, the display would act normally, but for other areas, it would put false data, preventing observers from being able to quickly peak at a user's documents.

Comments

  • Reply 1 of 7
    One of the differentiating features of the HomePod is its ability to hear you very well, making it easy to give commands when not facing it, raising your voice or even needing to be in the same room; so I doubt the gaze detection feature would be a requirement to use “hey Siri”, but I agree that it could be useful in making requests without first having to speak the activation command. A bit like how “raise to speak” on the Apple Watch functions. 
    Such a feature would be useful for hands free AppleTV use or easier use of Siri on iPhones (which already happen to include gaze detection features.)
    edited December 2020 Beatswatto_cobra
  • Reply 2 of 7
    Whilst I give Apple far more credit, when it comes to privacy, than other manufacturers, the thought of having additional cameras in the home (my HomePod currently sits on the bedside table) gives me serious pause for thought.

    I would hope to see this rely fully on faceID-type tech, rather than a full-vision (AI-based) camera system, else I believe I would be too worried to have one in more personal areas.
    edited December 2020
  • Reply 3 of 7
    Afganblues Afganblues Posts: 10unconfirmed, member
    Absolutely useless for people with low vision, or who are blind, or the HomePod is out of the line of sight,. There is one simple solution that I do not understand why Apple doesn’t implement. Hey HomePod or hey Siri pod or whatever the hell else you want to use as an alert phrase. We could even have custom FrasesWhy couldn’t the person program HomePod to respond to, let’s say something ridiculous like rumplestillskin.  Remember the reports of how angry Steve Jobs was over mobile me?, I’m sure he would be livid over this HomePod, iPhone, watch Siri differentiation issue. What really sucks that if I’m in a room with my  HomePod, phone and watch that I do not know where Siri will come from, especially when it’s a personal request and the HomePod responds with I can’t do that. Steve Jobs is probably rolling over in his grave
    edited December 2020 williamlondon
  • Reply 4 of 7
    BeatsBeats Posts: 2,561member
    Seems dumb. Isn't the point that you DON'T have to look at it?

    One of the differentiating features of the HomePod is its ability to hear you very well, making it easy to give commands when not facing it, raising your voice or even needing to be in the same room; so I doubt the gaze detection feature would be a requirement to use “hey Siri”, but I agree that it could be useful in making requests without first having to speak the activation command. A bit like how “raise to speak” on the Apple Watch functions. 
    Such a feature would be useful for hands free AppleTV use or easier use of Siri on iPhones (which already happen to include gaze detection features.)

    Great idea and hope this is the case.

    Absolutely useless for people with low vision, or who are blind, or the HomePod is out of the line of sight,. There is one simple solution that I do not understand why Apple doesn’t implement. Hey HomePod or hey Siri pod or whatever the hell else you want to use as an alert phrase. We could even have custom phrases. Why couldn’t the person program HomePod to respond to, let’s say something ridiculous like rumplestillskin.  Remember the reports of how angry Steve Jobs was over mobile me?, I’m sure he would be livid over this HomePod, iPhone, watch Siri differentiation issue. What really sucks that if I’m in a room with my  HomePod, phone and watch that I do not know where Siri will come from, especially when it’s a personal request and the HomePod responds with I can’t do that. Steve Jobs is probably rolling over in his grave

    I like that all devices are activated by ONE phrase. The problem is having the wrong device activate. @EsquireCats may be onto something.
  • Reply 5 of 7
    Absolutely useless for people with low vision, or who are blind, or the HomePod is out of the line of sight,. There is one simple solution that I do not understand why Apple doesn’t implement. Hey HomePod or hey Siri pod or whatever the hell else you want to use as an alert phrase. We could even have custom FrasesWhy couldn’t the person program HomePod to respond to, let’s say something ridiculous like rumplestillskin.  Remember the reports of how angry Steve Jobs was over mobile me?, I’m sure he would be livid over this HomePod, iPhone, watch Siri differentiation issue. What really sucks that if I’m in a room with my  HomePod, phone and watch that I do not know where Siri will come from, especially when it’s a personal request and the HomePod responds with I can’t do that. Steve Jobs is probably rolling over in his grave
    I'm not sure what differentiation issues you are referring to.  We have 8 x HomePods around the house, we both have iPhones & iPads that are on us or laying around, and 3-4 MacBooks on with the lids open, and both wear Apple Watches.  A HomePod ALWAYS answers, as they should, unless we have another device awake & actively using it.. Maybe you have a home WiFi or Bluetooth issue with interference, etc.?  The devices must be able to connect and announce to each other, who's handling the request.  There could also be Apple ID account issues, which is what ties all the devices together.  Probably an Apple ID/registration issue, or communications issue.
    watto_cobra
  • Reply 6 of 7
    I won't begin to say that HomePods are perfect. They're not - and I think the biggest problem (that we encounter) is that they seem to work better with a slight pause after "Hey Siri." My wife forgets that and it frustrates me because I know it's not going to work. But then again, about 50% of the time it does work for her. (As opposed to over 90% of the time if you add that ever-so-slight pause after "Siri.")

    Point being: if a visual clue improved upon that for her, I can't imagine it being a bad thing.


    edited December 2020 watto_cobra
  • Reply 7 of 7
    MacProMacPro Posts: 19,405member
    Here's looking at you Siri.
    watto_cobra
Sign In or Register to comment.