Apple wants to bring Zoom-like background replacement to 'Apple Glass'

Posted:
in General Discussion
Future "Apple Glass," or other such Apple headsets, may be able to replace a background with any image chosen by the wearer.

A single, solid-color background could be replaced by AR images through
A single, solid-color background could be replaced by AR images through "Apple Glass"


Apple has been investigating how to make an "Apple Glass" style headset perform chroma keying, or replacing of solid color backgrounds with Apple AR images. In most cases, the aim is for this processing to be done by the headset so that the result is quick enough that delays don't exacerbate motion sickness.

"Low latency chroma keying embedded in a head-mounted display for mixed reality," is a US patent application that was filed in February 2020 but only revealed this week. Since that filing, we've all experienced the coronavirus lockdown and its concomitant explosion in video conferencing.

Consequently, we're all now familiar with the idea of replacing the backgrounds of our offices and homes with other images, such as logos or beach scenes. We're also familiar with how rarely these work well, especially if the speaker moves around.

That's in part because even when not done very effectively, such background replacement takes significant processing power. Until a July 2020 update, Microsoft's Skype for iOS couldn't even blur backgrounds the way a Mac or PC could.

Such blurring is one complex process, but part of it involves determining where a person's head ends and their background begins. Chroma key makes this substantially easier because the person is positioned in front of a single solid-color backdrop. It's the same green- or blue-screen process used in movies.

When it's very apparent what part of a video conference frame is the person speaking and what is their background, a system like Zoom or Skype can more easily replace the background. Those are 2D systems, though, and "Apple Glass" aims to do it in 3D so that backgrounds can be replaced when the wearer is walking around, rather than just looking at a flat screen.

Left: a figure stands in front of a solid-colour wall. Right: the headset replaces the background
Left: a figure stands in front of a solid-colour wall. Right: the headset replaces the background


Apple's patent application is therefore simultaneously about making a 3D stereoscopic version of background replacement for Apple AR, and about making it work fast enough. Part of that has to do with the processing power of the headset, but it's also to do with the speed of returning data to the user's view.

"[What is needed is] a wearable immersive head-mounted display (HMD) doing real-time color-keying using embedded stereo RGB cameras to capture the environment," says the application, "and a display system to show real-world images augmented with virtual content."

"The system runs at a high frame rate (typically 75 fps or more)," it continues, "and it achieves a latency of less than one frame by capturing images and formatting them for display in the HMD itself."

So what Apple's proposal says is that a head-mounted display has "camera sensors to perform chroma keying in a mixed reality context," and to do it with a minimum of delay, or latency, for the wearer. "Low latency is achieved by embedding the processing in the HMD itself, specifically, format camera images, detect the selected color range and make a composite with the virtual content."

"Low latency is a key element required to achieve great immersion (often called "Presence")," continues the application.

Detail showing an example headset view before images have been replaced
Detail showing an example headset view before images have been replaced


Making this more complex is that such a system would actually have to create two images. "The system achieves mixed reality by having a user look at a display through wide angle lenses (FOV is typically 110.degree. or more)," it says. "Two cameras located behind the HMD display capture the environment from viewpoints located a few centimetres in front of each eye."

The application does not address the issue of where the replacement images come from. In a Zoom or Skype conference, it would be the choice of the speaker who has elected to present against a different background.

In the case of this application, there's no detail of how a speaker standing in front of an "Apple Glass" wearer would indicate what image would replace their blue or green screen. However, the patent does say that the practical applications include the use of games, videos and virtual objects."

The application is credited to seven inventors. They include Vincent Chapdelaine-Couture, whose previous related work includes a patent regarding "omnistereo imaging." And Bertrand Nepveu who is the sole credited inventor on a patent regarding digitally processing a video signal.

Detail showing an example headset view when images have been replaced
Detail showing an example headset view when images have been replaced


Separately, Apple has previously applied for patents including one for automatically blurring background or foreground images in video conferences.

Comments

  • Reply 1 of 7
    SpamSandwichSpamSandwich Posts: 33,408member
    It certainly won’t be called “Apple Glass” since “Google Glass” exists. They’ll be “Apple Glasses” if they’re anything.
  • Reply 2 of 7
    Whenever I hear people talking about replacing a background in Zoom my first thought is, “Just like iChat from 15 years ago!”

    I’m not sure I like the idea of replacing the background that is directly in front of me while wearing glasses. I don’t mind overlays but, in general, I want to see what’s in front of me. Changing the background during a videos conference is similar but different, considering I’m watching my display. 
    Beatswatto_cobra
  • Reply 3 of 7
    SpamSandwichSpamSandwich Posts: 33,408member
    Whenever I hear people talking about replacing a background in Zoom my first thought is, “Just like iChat from 15 years ago!”

    I’m not sure I like the idea of replacing the background that is directly in front of me while wearing glasses. I don’t mind overlays but, in general, I want to see what’s in front of me. Changing the background during a videos conference is similar but different, considering I’m watching my display. 
    Just watch out for that open manhole in the street... oh, too late.
  • Reply 4 of 7
    DAalsethDAalseth Posts: 1,724member
    Whenever I hear people talking about replacing a background in Zoom my first thought is, “Just like iChat from 15 years ago!”

    I’m not sure I like the idea of replacing the background that is directly in front of me while wearing glasses. I don’t mind overlays but, in general, I want to see what’s in front of me. Changing the background during a videos conference is similar but different, considering I’m watching my display. 
    Just watch out for that open manhole in the street... oh, too late.
    I was about to ask if I was the only one that saw a problem with this. I mean fine if you are standing still, but if you are moving I can see this hiding hazards. I can see someone running on a virtual beach in Tahiti, but encountering manholes, trees, traffic, angry dogs, etc.
    watto_cobra
  • Reply 5 of 7
    BeatsBeats Posts: 2,529member
    Scary tech. Reminds me of a Black Mirror episode where they would punish people by making them unable to contact people in pubilc.

    DAalseth said:
    Whenever I hear people talking about replacing a background in Zoom my first thought is, “Just like iChat from 15 years ago!”

    I’m not sure I like the idea of replacing the background that is directly in front of me while wearing glasses. I don’t mind overlays but, in general, I want to see what’s in front of me. Changing the background during a videos conference is similar but different, considering I’m watching my display. 
    Just watch out for that open manhole in the street... oh, too late.
    I was about to ask if I was the only one that saw a problem with this. I mean fine if you are standing still, but if you are moving I can see this hiding hazards. I can see someone running on a virtual beach in Tahiti, but encountering manholes, trees, traffic, angry dogs, etc.

    I thought it was for special applications like gaming, museums etc. and not something the user can just turn on for the fun of it.
    martinp13
  • Reply 6 of 7
    BeatsBeats Posts: 2,529member
    Whenever I hear people talking about replacing a background in Zoom my first thought is, “Just like iChat from 15 years ago!”

    I’m not sure I like the idea of replacing the background that is directly in front of me while wearing glasses. I don’t mind overlays but, in general, I want to see what’s in front of me. Changing the background during a videos conference is similar but different, considering I’m watching my display. 

    Like how kids praise Snapchat for filters and effects when iChat was doing it a decade earlier.

    iChat was way ahead of time. I wish Apple would have taken it more seriously. I remember waiting for Photobooth for iPhone back in 2012. Never came.
    watto_cobra
  • Reply 7 of 7
    chasmchasm Posts: 2,391member
    Firstly, Zoom-like anything on AR/VR glasses is a terrible idea if you're moving. I would expect Apple would -- as with CarPlay -- make it impossible to do certain distracting tasks when wearing them while moving/driving.

    Second, whatever kind of background-shifting trick they might someday introduce, please baby Jesus don't make it anything like Zoom's virtual backgrounds -- they are the worst I've ever seen, even worse than iChat's background all those years ago! The slightest movement on the part of the speaker or others can cause whole limbs, people, and and portions of faces to disappear. I refer to the effect as the "uncanny valley inducer."

    If you have to hide your messy background with a virtual one, do Zoom and yourself a huge favour and put up a solid-colour blanket behind you to make the virtual background work better. And of course don't wear the same colour unless you are intentionally going for the "floating head" effect! Ha!
    watto_cobra
Sign In or Register to comment.