Apple wants to bring Zoom-like background replacement to 'Apple Glass'
Future "Apple Glass," or other such Apple headsets, may be able to replace a background with any image chosen by the wearer.
A single, solid-color background could be replaced by AR images through "Apple Glass"
Apple has been investigating how to make an "Apple Glass" style headset perform chroma keying, or replacing of solid color backgrounds with Apple AR images. In most cases, the aim is for this processing to be done by the headset so that the result is quick enough that delays don't exacerbate motion sickness.
"Low latency chroma keying embedded in a head-mounted display for mixed reality," is a US patent application that was filed in February 2020 but only revealed this week. Since that filing, we've all experienced the coronavirus lockdown and its concomitant explosion in video conferencing.
Consequently, we're all now familiar with the idea of replacing the backgrounds of our offices and homes with other images, such as logos or beach scenes. We're also familiar with how rarely these work well, especially if the speaker moves around.
That's in part because even when not done very effectively, such background replacement takes significant processing power. Until a July 2020 update, Microsoft's Skype for iOS couldn't even blur backgrounds the way a Mac or PC could.
Such blurring is one complex process, but part of it involves determining where a person's head ends and their background begins. Chroma key makes this substantially easier because the person is positioned in front of a single solid-color backdrop. It's the same green- or blue-screen process used in movies.
When it's very apparent what part of a video conference frame is the person speaking and what is their background, a system like Zoom or Skype can more easily replace the background. Those are 2D systems, though, and "Apple Glass" aims to do it in 3D so that backgrounds can be replaced when the wearer is walking around, rather than just looking at a flat screen.
Left: a figure stands in front of a solid-colour wall. Right: the headset replaces the background
Apple's patent application is therefore simultaneously about making a 3D stereoscopic version of background replacement for Apple AR, and about making it work fast enough. Part of that has to do with the processing power of the headset, but it's also to do with the speed of returning data to the user's view.
"[What is needed is] a wearable immersive head-mounted display (HMD) doing real-time color-keying using embedded stereo RGB cameras to capture the environment," says the application, "and a display system to show real-world images augmented with virtual content."
"The system runs at a high frame rate (typically 75 fps or more)," it continues, "and it achieves a latency of less than one frame by capturing images and formatting them for display in the HMD itself."
So what Apple's proposal says is that a head-mounted display has "camera sensors to perform chroma keying in a mixed reality context," and to do it with a minimum of delay, or latency, for the wearer. "Low latency is achieved by embedding the processing in the HMD itself, specifically, format camera images, detect the selected color range and make a composite with the virtual content."
"Low latency is a key element required to achieve great immersion (often called "Presence")," continues the application.
Detail showing an example headset view before images have been replaced
Making this more complex is that such a system would actually have to create two images. "The system achieves mixed reality by having a user look at a display through wide angle lenses (FOV is typically 110.degree. or more)," it says. "Two cameras located behind the HMD display capture the environment from viewpoints located a few centimetres in front of each eye."
The application does not address the issue of where the replacement images come from. In a Zoom or Skype conference, it would be the choice of the speaker who has elected to present against a different background.
In the case of this application, there's no detail of how a speaker standing in front of an "Apple Glass" wearer would indicate what image would replace their blue or green screen. However, the patent does say that the practical applications include the use of games, videos and virtual objects."
The application is credited to seven inventors. They include Vincent Chapdelaine-Couture, whose previous related work includes a patent regarding "omnistereo imaging." And Bertrand Nepveu who is the sole credited inventor on a patent regarding digitally processing a video signal.
Detail showing an example headset view when images have been replaced
Separately, Apple has previously applied for patents including one for automatically blurring background or foreground images in video conferences.
A single, solid-color background could be replaced by AR images through "Apple Glass"
Apple has been investigating how to make an "Apple Glass" style headset perform chroma keying, or replacing of solid color backgrounds with Apple AR images. In most cases, the aim is for this processing to be done by the headset so that the result is quick enough that delays don't exacerbate motion sickness.
"Low latency chroma keying embedded in a head-mounted display for mixed reality," is a US patent application that was filed in February 2020 but only revealed this week. Since that filing, we've all experienced the coronavirus lockdown and its concomitant explosion in video conferencing.
Consequently, we're all now familiar with the idea of replacing the backgrounds of our offices and homes with other images, such as logos or beach scenes. We're also familiar with how rarely these work well, especially if the speaker moves around.
That's in part because even when not done very effectively, such background replacement takes significant processing power. Until a July 2020 update, Microsoft's Skype for iOS couldn't even blur backgrounds the way a Mac or PC could.
Such blurring is one complex process, but part of it involves determining where a person's head ends and their background begins. Chroma key makes this substantially easier because the person is positioned in front of a single solid-color backdrop. It's the same green- or blue-screen process used in movies.
When it's very apparent what part of a video conference frame is the person speaking and what is their background, a system like Zoom or Skype can more easily replace the background. Those are 2D systems, though, and "Apple Glass" aims to do it in 3D so that backgrounds can be replaced when the wearer is walking around, rather than just looking at a flat screen.
Left: a figure stands in front of a solid-colour wall. Right: the headset replaces the background
Apple's patent application is therefore simultaneously about making a 3D stereoscopic version of background replacement for Apple AR, and about making it work fast enough. Part of that has to do with the processing power of the headset, but it's also to do with the speed of returning data to the user's view.
"[What is needed is] a wearable immersive head-mounted display (HMD) doing real-time color-keying using embedded stereo RGB cameras to capture the environment," says the application, "and a display system to show real-world images augmented with virtual content."
"The system runs at a high frame rate (typically 75 fps or more)," it continues, "and it achieves a latency of less than one frame by capturing images and formatting them for display in the HMD itself."
So what Apple's proposal says is that a head-mounted display has "camera sensors to perform chroma keying in a mixed reality context," and to do it with a minimum of delay, or latency, for the wearer. "Low latency is achieved by embedding the processing in the HMD itself, specifically, format camera images, detect the selected color range and make a composite with the virtual content."
"Low latency is a key element required to achieve great immersion (often called "Presence")," continues the application.
Detail showing an example headset view before images have been replaced
Making this more complex is that such a system would actually have to create two images. "The system achieves mixed reality by having a user look at a display through wide angle lenses (FOV is typically 110.degree. or more)," it says. "Two cameras located behind the HMD display capture the environment from viewpoints located a few centimetres in front of each eye."
The application does not address the issue of where the replacement images come from. In a Zoom or Skype conference, it would be the choice of the speaker who has elected to present against a different background.
In the case of this application, there's no detail of how a speaker standing in front of an "Apple Glass" wearer would indicate what image would replace their blue or green screen. However, the patent does say that the practical applications include the use of games, videos and virtual objects."
The application is credited to seven inventors. They include Vincent Chapdelaine-Couture, whose previous related work includes a patent regarding "omnistereo imaging." And Bertrand Nepveu who is the sole credited inventor on a patent regarding digitally processing a video signal.
Detail showing an example headset view when images have been replaced
Separately, Apple has previously applied for patents including one for automatically blurring background or foreground images in video conferences.
Comments
I’m not sure I like the idea of replacing the background that is directly in front of me while wearing glasses. I don’t mind overlays but, in general, I want to see what’s in front of me. Changing the background during a videos conference is similar but different, considering I’m watching my display.
I thought it was for special applications like gaming, museums etc. and not something the user can just turn on for the fun of it.
Like how kids praise Snapchat for filters and effects when iChat was doing it a decade earlier.
iChat was way ahead of time. I wish Apple would have taken it more seriously. I remember waiting for Photobooth for iPhone back in 2012. Never came.