'Foveated' display may boost Apple AR headset refresh rates

Posted:
in Future Apple Hardware
VR and AR headsets could offer high refresh rates in their screens, Apple proposes, by using a 'foveated display' and an eye tracking system to monitor the gaze of the user and to optimize rendering to focus only where the user is actively looking on a display.




One of the key elements of a virtual reality or augmented reality headset is the use of displays with a high refresh rate. Screens that update quicker and allow for faster responses to user motion can minimize the potential of nausea from not updating the screen to match movements, as well as further selling the illusion of the virtual environment or digital object being placed in the world.

One of the problems with advancing the technology used for headset displays is that there is an expectation of using higher resolution screens. By adding more pixels to a display, that means there's more elements that have to be updated each refresh, and more data that needs to be created by the host device in rendering the scene.

In a patent application published by the US Patent and Trademark Office on Thursday, the filing for a "Foveated Display" aims to solve these problems by offering two different streams of data for a display to use, consisting of high-resolution and low-resolution imagery.

The core concept of a foveated display is that the screen does not have to use high-resolution imagery for the entire screen, only whatever the user is looking at. If the position can be determined, it is possible for a display to show the resource-heavy higher-resolution picture in the user's direct view, then use the lightweight low-resolution data for the remainder of the screen.

A diagram explaining how a display's system typically operates for refreshes
A diagram explaining how a display's system typically operates for refreshes


As the rest of the display would take advantage of the user's peripheral vision, which doesn't require detail, this can significantly cut down on the amount of work that needs to be accomplished each time the display needs refreshing.

In Apple's solution, a gaze-tracking system is used to find out the point on the screen the user's eyes are trained on. Knowing this data, a graphics processing unit then renders a high-resolution image for part of the scene where the user is looking, as well as a low-resolution version for the remainder of the picture.

Using timing controller circuitry and column driver circuitry, the former can provide the image data to the latter, which then implements the changes to the display. This circuitry is also used to switch between two buffers, providing high-resolution data and low-resolution data, with each employed for their respective regions of the display.

The use of interpolation and filter circuitry could be used to alter the pixel data before it is applied, such as in areas where low-resolution data is used alongside the high-resolution version to even out any apparent seams. Two-dimensional spatial filters could also be applied to the low-resolution data buffer.

An illustration of how varied resolutions could be assembled on a display based on the user looking in the middle of the screen.
An illustration of how varied resolutions could be assembled on a display based on the user looking in the middle of the screen.


Apple files numerous patent applications with the USPTO on a weekly basis, and while it does show areas of interest for the company, it doesn't guarantee that Apple will be using the technology in a future product or service.

Apple has been long rumored to be working on a VR or AR headset, one which could launch as early as 2020 or 2021 according to some analysts. The device, possibly under the codename "T288," may feature 8K-resolution eyepieces and use the 60-gigahertz WiGig wireless networking system.

There have also been a number of patents and applications relating to headsets that have surfaced over the years, including for thermal regulation, fitting the headset to the user's head, and even for glasses that hold an iPhone as a display.

Related earlier filings for the latest discovery include a previous "Predictive, Foveated Virtual Reality System" that used a similar method of different-resolution video feeds and selective rendering to minimize latency to the user. Apple has also explored a variety of eye tracking systems, including some using a "hot mirror" allowing the components to be close to the user's face rather than further away, making the headset more comfortable to wear for longer periods.

Comments

  • Reply 1 of 13
    Lots of prior art for that one. Using multiple streams at different resolutions that track with your eyes motion with prediction is the very definition of foviated display for VR and AR.
  • Reply 2 of 13
    Lots of prior art for that one. Using multiple streams at different resolutions that track with your eyes motion with prediction is the very definition of foviated display for VR and AR.
    Right.  The patent is for methods of implementing foviation, not for the concept itself.  As is the case with every patent.
    StrangeDaysbeowulfschmidtbestkeptsecret
  • Reply 3 of 13
    SpamSandwichSpamSandwich Posts: 31,191member
    All well and good, but until AR glasses are as light and unobtrusive as a regular pair of glasses (I’m confident such a thing WILL eventually be possible) there will be strong resistance to widespread adoption of such a product.
  • Reply 4 of 13
    racerhomie3racerhomie3 Posts: 1,142member
    Will the technology Apple is developing damage our eyesight?
  • Reply 5 of 13
    georgie01georgie01 Posts: 248member
    Interesting and great idea if done seamlessly. Most people don’t realise only a relatively small part of their field of view is sharp. The brain is extremely good at fooling us into thinking we’re seeing things we’re not. There’s even a small gap in our vision due to the optic nerve and the brain cleverly makes up something to put there so that we don’t notice. Or thinking we can see a full range of colour in our periphery which again is just our brain fooling us.
  • Reply 6 of 13
    gatorguygatorguy Posts: 20,748member
    Will the technology Apple is developing damage our eyesight?
    It wouldn't be very consumer-friendly if it did, not great for PR either, and be a just a tad expensive on the legal side, even ignoring the dearth of sales if it were harmful. That should answer the question. 
    edited June 13 muthuk_vanalingamterrence1019
  • Reply 7 of 13
    All well and good, but until AR glasses are as light and unobtrusive as a regular pair of glasses (I’m confident such a thing WILL eventually be possible) there will be strong resistance to widespread adoption of such a product.
    It will also need a good use case. I’m not saying there won’t be any but right now I don’t see anything that’s 1) obvious and 2) would be worth wearing glasses all the time for.

    For example, I can absolutely see how having walking directions overlayed on what I was seeing in front of me when I was going somewhere unfamiliar would be worthwhile. But, for me, that doesn’t happen often enough to warrant carrying another piece of technology around. Especially when I can just glance at my watch fairly inconspicuously and get a similar result (navigation).

    Some people seem to think that AR glasses would replace their phone. I could see that being possible but again, who wants to wear glasses all the time if they aren’t needed? I carry my phone in my pocket pretty much all day and it is easily accessible. I don’t wear glasses at all (yet). The thought of wearing glasses the same amount of time as I have my phone with me is not particularly appealing, and taking them out to put them on to use and then promptly putting them away sounds cumbersome.
  • Reply 8 of 13
    StrangeDaysStrangeDays Posts: 7,989member
    Lots of prior art for that one. Using multiple streams at different resolutions that track with your eyes motion with prediction is the very definition of foviated display for VR and AR.
    Gosh, we’d better let Apple know. They don’t know what we know! Their patents could be in jeopardy!
  • Reply 9 of 13
    StrangeDaysStrangeDays Posts: 7,989member

    All well and good, but until AR glasses are as light and unobtrusive as a regular pair of glasses (I’m confident such a thing WILL eventually be possible) there will be strong resistance to widespread adoption of such a product.
    There is additional lead way for private use. 
  • Reply 10 of 13
    StrangeDaysStrangeDays Posts: 7,989member

    Will the technology Apple is developing damage our eyesight?
    I can’t tell if you’re serious or not, which is kinda sad. 
  • Reply 11 of 13
    SpamSandwichSpamSandwich Posts: 31,191member
    All well and good, but until AR glasses are as light and unobtrusive as a regular pair of glasses (I’m confident such a thing WILL eventually be possible) there will be strong resistance to widespread adoption of such a product.
    It will also need a good use case. I’m not saying there won’t be any but right now I don’t see anything that’s 1) obvious and 2) would be worth wearing glasses all the time for.

    For example, I can absolutely see how having walking directions overlayed on what I was seeing in front of me when I was going somewhere unfamiliar would be worthwhile. But, for me, that doesn’t happen often enough to warrant carrying another piece of technology around. Especially when I can just glance at my watch fairly inconspicuously and get a similar result (navigation).

    Some people seem to think that AR glasses would replace their phone. I could see that being possible but again, who wants to wear glasses all the time if they aren’t needed? I carry my phone in my pocket pretty much all day and it is easily accessible. I don’t wear glasses at all (yet). The thought of wearing glasses the same amount of time as I have my phone with me is not particularly appealing, and taking them out to put them on to use and then promptly putting them away sounds cumbersome.
    Here are a few real-world use cases I could imagine if the glasses were as light and unobtrusive as regular glasses: A virtual computer display could be projected to replace a physical display and a virtual TV that is a persistent image could replace a large screen television. And the glasses could also magnify small images so you’d have an instant magnifier. All useful possibilities.
  • Reply 12 of 13
    fastasleepfastasleep Posts: 2,975member
    Will the technology Apple is developing damage our eyesight?
    Clearly. 
  • Reply 13 of 13
    Will the technology Apple is developing damage our eyesight?
    Clearly. 

    Yes. If he cannot see that, his eyesight is already damaged!
Sign In or Register to comment.