Future version of Animoji might add voice, sound effects based on user expressions

Posted:
in iOS
Apple's Animoji may provide more ways to interact with the face-tracking animated characters, by playing appropriate sound effects for each avatar triggered when users either make specific expressions or say certain words during a recording session.




Launched alongside the iPhone X as a demonstration of what people can do with the TrueDepth camera system, Animoji quickly became a popular addition to iOS that spawned commercials and "Animoji Karaoke" videos across the Internet. While Apple has added more interactivity to the face-tracked creations, including tongue detection and user-created Memoji, it seems that Apple has more ideas of where Animoji can go.

A patent application filed on February 28 and published by the U.S. Patent and Trademark Office on November 22 for "Voice effects based on facial expressions" is all about how a video clip of a virtual avatar is created, but can be altered with special effects that the user can trigger. In short, Animoji but with added extra features.

During the recording phase of the Animoji message, the software captures the facial movements and audio from the subject. Usually, the face tracking is mapped directly to the character, with the user's movements mirrored as closely as possible, while the audio track is exactly what was picked up by the iPhone during the period.

According to the patent application, both the visual and audio elements are monitored for specific states, which can then be changed during the playback phase to something else. These states can include detected facial expressions, such as frowning, or even trigger words, which could trigger an effect.

Once triggered, the expression creates an effect that is applied to the character's visuals or the audio section. The character could go through a predefined animation or be altered in certain ways while the expression is being made, while accompanying speech could be pitch shifted or replaced entirely.

An illustrated basic example of changing a user's 'woof' speech to a recording of a dog bark
An illustrated basic example of changing a user's 'woof' speech to a recording of a dog bark


For example, a person using a dog avatar could say the word "bark," resulting in the playing of an audio file of a dog barking and altered mouth shapes on the character to match. A user frowning and growling could transform a pleasant dog character into an angry version, with a pitch-shifted or replaced growling sound.

The voice recording in its entirety could be replaced by a synthesized voice in some cases, with voice recognition detecting individual words, pitch, and cadence that can then be reproduced in a character-specific voice.

While the filing of a patent application is not a guarantee that it will appear in a future Apple product or service, while also serving as an indicator of where Apple's interests lie, in this case the concept has a good chance of being implemented in a future iOS update.

In the case of Animoji, it already takes advantage of the facial tracking elements of TrueDepth camera-equipped iPhones, making it feasible for Apple to add in emotion recognition. The voice recognition capabilities, as already demonstrated by the existence of Siri, also lends itself to the audio-tampering elements of the patent application.

Comments

  • Reply 1 of 12
    I wonder if there's going to be a fart sound... when you make "that" face.  :D
    rossb2
  • Reply 2 of 12
    ivanhivanh Posts: 236member
    Is it considered an innovation or a “wow” factor? It’s not even a nice-to-have feature. 👎
    dysamoria
  • Reply 3 of 12
    entropysentropys Posts: 1,374member
    I. Can’t. Wait.
    rossb2dysamoria
  • Reply 4 of 12
    I wonder if there's going to be a fart sound... when you make "that" face.  :D
    well they have an animoji that is a big turd already, so your suggestion is just one step further. classy.... Just what you want on a premium phone LOL
  • Reply 5 of 12
    although animoji seems like vapourware now, think about how this technology could evolve. Virtual assistants? AI bots answering calls for corporations?
    dewme
  • Reply 6 of 12
    entropysentropys Posts: 1,374member
    If I am attracted to my wife’s Animoji, am I being unfaithful?
  • Reply 7 of 12
    I get why Apple added this feature - it sells phones to certain segments. But it is so frustrating to see effort put into this type of feature rather than improve/fix Siri.
    beowulfschmidtdysamoria
  • Reply 8 of 12
    dewmedewme Posts: 1,798member
    What's the end game here? Are we slowly moving towards a universal hieroglyphic language standard? Not trying to be silly here. We already have a few instances where emoji-like visuals are used for concrete communication, like crosswalks and warning signs for certain hazards, e.g., falling over cliff, fingers crushed in machinery gears, etc. If so this would require individuals participating in emoji based communication to always have a device with visual rendering capability within reach, which would be a boon to companies like Apple.Emoji is unlikely to be adopted as a primary language but it could be a universal secondary language and a bridge between existing languages. I guess we'll know where this is going if or when a standard set of syntax rules for Emoji are published, e.g., does the poop pile go before or after the cartoon human character?
  • Reply 9 of 12
    Mike WuertheleMike Wuerthele Posts: 3,715administrator
    Fatman said:
    I get why Apple added this feature - it sells phones to certain segments. But it is so frustrating to see effort put into this type of feature rather than improve/fix Siri.
    I don't think the Animoji crew of developers are the same folks that work on Siri.
  • Reply 10 of 12
    dysamoriadysamoria Posts: 1,858member
    Imagine where this might go: some day we might have face and voice communication that looks photographic and sounds like us!

    Oh wait. We HAVE that now.

    This crap just keeps wasting resources and bloating the OS. “Please keep buying the same product again and again every year!”, pleads Apple, “our shareholders expect perpetual growth!”
  • Reply 11 of 12
    FatmanFatman Posts: 180member
    Fatman said:
    I get why Apple added this feature - it sells phones to certain segments. But it is so frustrating to see effort put into this type of feature rather than improve/fix Siri.
    I don't think the Animoji crew of developers are the same folks that work on Siri.
    I guess my point is, maybe there shouldn't be an animoji team, until they nail the more important software features & hardware functions. I'd rather have Apple invest in resources for Siri development, Apple maps improvements, better battery life, or next gen displays, than having a team working on tracking and animating people's tongues!
  • Reply 12 of 12
    Its the next logical approach. Tweaking your voice would add fun to the fun characters, lets face it, not everyone is very good at impersonations so this could be a quite feature.
Sign In or Register to comment.