Apple is teaching Siri how to read lips

in Future Apple Hardware

Future Apple devices may be able to use motion detection to read lips, and so trigger Siri without needing a microphone to constantly listen out for commands.

HAL 9000 background source: Warner Bros
HAL 9000 background source: Warner Bros

If you're old enough, the notion of Siri being able to read lips in any way has immediately and worryingly brought Arthur C. Clarke and Stanley Kubrick's "2001: A Space Odyssey" to mind. Hopefully if Apple is channeling that 1968 film, it is because the computer HAL 9000 had superb voice recognition skills.

In comparison, Siri has much more difficulty reliably and consistently understanding spoken commands, but to be fair it also hasn't yet tried to kill the crew of a spaceship. It's swings and balances.

Conceivably, though, giving Siri an extra aspect such as detecting mouth and head movements could improve its accuracy. A newly-revealed patent application called "Keyword Detection Using Motion Sensing," aims to do that -- but then something more.

"[Data] is received from a motion sensor, for instance, recording the motion of a user as the user utters a spoken input," says the patent application. "A determination is made whether a portion of the motion data matches reference data for a set of one or more words (e.g., a word or phrase)."

"Additionally, voice [only] control systems can result in false positive responses ," mentioned Apple, "if the audio sensor picks up ambient noise or speech from an unintended user."

The patent application details how mouth movements can be compared against previous data as Siri or a device attempts to find a match.

Detail from the patent showing how motion detection can be compared against previous data to determine what someone is saying
Detail from the patent showing how motion detection can be compared against previous data to determine what someone is saying

But this is not really for improving Siri, and it's not a sign that Apple is planning some devices without microphones. Instead, Apple proposes that such motion detection could mean being able to switch off the microphones that a device uses to constantly listen for "Siri," or "Hey, Siri."

"[Continuously] detecting and processing audio data expends power and processing capacity even when the user is not actively using voice control," says Apple.

"When a user speaks, the user's mouth, face, head, and neck move and vibrate," it continues. "Motion sensors such as accelerometers and gyroscopes can detect these motions, while expending relatively little power compared to audio sensors such as microphones."

Detecting motion now and comparing it to previous records seems clearly able to work when what's being said is "Hey, Siri," or some other regular command. like "Next track." When the spoken command is less common, such as "Hey, Siri, open the pod bay doors," then surely motion detection won't work.

But as long as motion detection is fast enough, spotting that a user has said "Siri" should mean the device being able to turn on the microphones in time to catch the rest vocally.

Other than referring to accelerometers and gyroscopes, Apple's patent application doesn't spend much time discussing the devices that could be used to implement this proposal.

However, it is lip reading by motion detection, rather than through cameras and line of sight. So, especially in conjunction with an iPhone, this motion detection could theoretically work with AirPods as well as, for instance, Apple Vision Pro.

This patent application is credited to two inventors, including Madhu Chinthakunta. Chinthakunta's previous work for Apple includes a patent for having Siri automatically make arrangements and calls on your behalf.

Read on AppleInsider


  • Reply 1 of 10
    mayflymayfly Posts: 385member
    Hope it will only read the user's lips.
  • Reply 2 of 10
    dewmedewme Posts: 5,282member
    Ventriloquists are screwed.
  • Reply 3 of 10
    gatorguygatorguy Posts: 24,104member
    dewme said:
    Ventriloquists are screwed.
    He's Siri-ous?
  • Reply 4 of 10

    I suspect that Apple AI might be more advanced than what people might think.

  • Reply 5 of 10
    gatorguygatorguy Posts: 24,104member
    lorca2770 said:

    I suspect that Apple AI might be more advanced than what people might think.

    They're ALL more advanced that we realize. For the time-being the capabilities of the current AI systems are being purposely limited.
  • Reply 6 of 10
    Well - perhaps Siri will then finally better understand what I SAY....

    Is it just me, or has Siri become more stupid over time? Please don't answer if you're American or British, because Siri was designed for you and therefore works best for you.

    Try Siri in another language while driving, and pray that it will return with another answer than "I did not understand that", or "Look what I found on the internet" (prompting you to do all the unsafe things that Apple professes to keep you from doing while you're on the Autobahn)... Or play Parov Stelar instead of the requested Beethoven.

    I vaguely remember that I was genuinely impressed with Siri's deep "understanding" when it first became available in Europe. I am quite sure that it has lost much of that depth.

    Now, **I** have to watch YouTube videos or take online courses to learn what questions I can safely ask / how to ask them / and what things will not work with Siri.
    Some PDA.... Honestly, the time it takes Siri to understand and do what I want by far surpasses the time to just pick up the phone and use my finger.

    Perhaps it will help if it can not only hear my voice, but see my lips as well, but I won't hold my breath.
  • Reply 7 of 10
    CiaranFCiaranF Posts: 23member
    Hey , I think it would be better if you figured out how Siri responded to our voice first and understood what we were telling it. Baby steps first. She’s the dumbest assistant out there at the moment, please don’t dig the hole deeper by trying to get her to read lips when there’s a ton of work to be done with her trying to listen to our commands. 

  • Reply 8 of 10
    I’m sorry Dave, I’m afraid I can’t do that.
  • Reply 9 of 10
    DracoDraco Posts: 34member
    Siri has been handicapped by Apple's policy of maintaining user privacy. They will have the same problem with implementing AI. 
  • Reply 10 of 10
    mattinozmattinoz Posts: 2,279member
    It isn't reading lips thou. it is reading the ears of the user.
    I think that is actually rather impressive if it even remoting works 
Sign In or Register to comment.