Apple refining video compression, haptic feedback for AR and VR headsets
Apple is continuing to develop and refine technology for head-mounted devices like "Apple Glass," including video compression methods and directional haptic feedback.
Credit: Microsoft
The Cupertino tech giant is said to be working on two distinct HMDs, including a mixed-reality headset leveraging Apple VR technology and the aforementioned Apple AR device. On Tuesday, Apple was granted a pair of patents that could apply to either.
In the first patent, titled "Electronic devices with directional haptic output," Apple outlines a method that could provide haptic feedback "in a given direction relative to a device housing surface such as a sidewall surface or other device surface."
Firstly, Apple outlines some of the problems with haptic feedback, including output that is too weak or provides an undesired sensation. Additionally, it says some systems may provide haptic output that's not applied to an appropriate location on the body of a user. Apple also notes a haptic output device may be too bulky, which is an important factor in HMDs.
Although seemingly not specific to an HMD, Apple does note that the method of operating the system could use both a head-mounted device and a cellular phone together.
That method would comprise a motion sensor gathering user input to move a virtual object relative to other virtual objects in a display. When a user applies a certain amount of force or the object interacts with other virtual objects around, the system would apply directional feedback providing "a sensation of resistance to further movement of the portable electronic device."
Apple first applied for this patent in July 2017. It credits Paul X. Wang and Michael Y. Cheung as its inventors. The former inventor has been listed on haptic systems for glass keyboards and the Apple Pencil.
The second patent, titled "Video compression methods and apparatus," more clearly outlines a method for a head-mounted device. This patent focuses on a system that's based on a "mixed reality system including a head-mounted display and a base station."
As Apple notes, data collected by an HMD sensor suite could be transferred to the base station via wired or wireless connection.
From there, "a rendering engine renders frames including virtual content based in part on the sensor information, and an encoder compresses the frames according to an encoding protocol before sending the frames to the HMD over the connection. Instead of using a previous frame to estimate motion vectors in the encoder, motion vectors from the HMD and the rendering engine are input to the encoder and used in compressing the frame."
Video compression will be an important part of head-mounted devices, particularly when rumors suggest that they'll use a suite of cameras and ultra-high-resolution 8K displays for video output.
The system Apple outlines here envisions a method that could reduce the time it takes to encode rendered frames on a base station before transmitting them to the HMD.
Additionally, Apple notes that there could be times when specific frames aren't properly transmitted to the HMD. In these cases Apple says the AR or VR device "may synthesize a frame from a previous frame using the motion vectors," ensuring a smooth video output.
These specific frames could be both a user's actual surrounding environment, or any number of virtual objects and animations. Together, both provide a mixed-reality, or MR, effect.
The video compression patent, granted on Feb. 9 and first applied for on May 30, 2017, credits Geoffrey Stahl and Avi Bar-Zeev as its inventors. The latter inventor, notably, is the creator of Microsoft's Hololens and has been named on Apple VR-related patents. He left Apple in 2019.
Apple files numerous patents on a weekly basis, and many of the technologies described within them don't end up in a consumer product. Patents and patent applications also don't give any indication of when a feature or system might make it to market.
Credit: Microsoft
The Cupertino tech giant is said to be working on two distinct HMDs, including a mixed-reality headset leveraging Apple VR technology and the aforementioned Apple AR device. On Tuesday, Apple was granted a pair of patents that could apply to either.
In the first patent, titled "Electronic devices with directional haptic output," Apple outlines a method that could provide haptic feedback "in a given direction relative to a device housing surface such as a sidewall surface or other device surface."
Firstly, Apple outlines some of the problems with haptic feedback, including output that is too weak or provides an undesired sensation. Additionally, it says some systems may provide haptic output that's not applied to an appropriate location on the body of a user. Apple also notes a haptic output device may be too bulky, which is an important factor in HMDs.
Although seemingly not specific to an HMD, Apple does note that the method of operating the system could use both a head-mounted device and a cellular phone together.
That method would comprise a motion sensor gathering user input to move a virtual object relative to other virtual objects in a display. When a user applies a certain amount of force or the object interacts with other virtual objects around, the system would apply directional feedback providing "a sensation of resistance to further movement of the portable electronic device."
Apple first applied for this patent in July 2017. It credits Paul X. Wang and Michael Y. Cheung as its inventors. The former inventor has been listed on haptic systems for glass keyboards and the Apple Pencil.
The second patent, titled "Video compression methods and apparatus," more clearly outlines a method for a head-mounted device. This patent focuses on a system that's based on a "mixed reality system including a head-mounted display and a base station."
As Apple notes, data collected by an HMD sensor suite could be transferred to the base station via wired or wireless connection.
From there, "a rendering engine renders frames including virtual content based in part on the sensor information, and an encoder compresses the frames according to an encoding protocol before sending the frames to the HMD over the connection. Instead of using a previous frame to estimate motion vectors in the encoder, motion vectors from the HMD and the rendering engine are input to the encoder and used in compressing the frame."
Video compression will be an important part of head-mounted devices, particularly when rumors suggest that they'll use a suite of cameras and ultra-high-resolution 8K displays for video output.
The system Apple outlines here envisions a method that could reduce the time it takes to encode rendered frames on a base station before transmitting them to the HMD.
Additionally, Apple notes that there could be times when specific frames aren't properly transmitted to the HMD. In these cases Apple says the AR or VR device "may synthesize a frame from a previous frame using the motion vectors," ensuring a smooth video output.
These specific frames could be both a user's actual surrounding environment, or any number of virtual objects and animations. Together, both provide a mixed-reality, or MR, effect.
The video compression patent, granted on Feb. 9 and first applied for on May 30, 2017, credits Geoffrey Stahl and Avi Bar-Zeev as its inventors. The latter inventor, notably, is the creator of Microsoft's Hololens and has been named on Apple VR-related patents. He left Apple in 2019.
Apple files numerous patents on a weekly basis, and many of the technologies described within them don't end up in a consumer product. Patents and patent applications also don't give any indication of when a feature or system might make it to market.