Apple's headset to use 3D operating system that looks like iOS

Posted:
in Future Apple Hardware edited February 2023
Apple's VR headset will feature a 3D operating system that will resemble iOS with hand and finger gesture recognition -- and can be used as a macOS display.

A render of a potential Apple headset [AppleInsider]
A render of a potential Apple headset [AppleInsider]


The long-rumored Apple AR headset could finally become a real product in 2023, but there are still many questions about how it will actually function. In a new breakdown of how the headset will work, Apple's attempt to take on Meta and others will rely on producing a great operating system experience, as well as high-specification hardware.

Driving the entire device will be an operating system that will have an iOS-like interface, according to Bloomberg. The operating system, xrOS, will be completely in 3D, and offer many of the same features as iOS and iPadOS, including a version of Safari, Photos, Mail, Messages, and Calendar.

The aim is to apparently offer an experience familiar to iPhone and iPad users.

To navigate the environment, hand, finger, and eye tracking will be used via several cameras and sensors, with gestures such as pinching their thumb and index finger used to trigger items. Eye tracking will allow users to select elements from a menu, without clumsily doing so with their hands.

Despite being a headset that covers the head, FaceTime will still be usable, with the headset realistically rendering a user's face and body in VR as an avatar. This would be wildly different from Meta's approach, which relies on cartoon-like avatars instead.

The realistic avatars will only be supported in one-on-one video chats due to the high amount of processing power required for it to work. For multiple people, additional users could be displayed as an icon or as Memoji.

A Digital Crown, like the Apple Watch version, will be used to switch users between AR and VR modes, so that they can gradually move between the two depending on how much of the real-world environment they need to see.

To further productivity, the headset will also be able to work as an external monitor for a Mac. The user's Mac display will be visible in VR, but control of the computer is still performed using a trackpad, mouse, or physical keyboard.

This sort of system is available elsewhere, including Meta's Horizon Workrooms, which also employs a passthrough system so you can access a Mac's keyboard and mouse.

To enter text into the system, users will still have an option to do it entirely in-headset. Technology is being made to allow users to type in midair with their hands on a virtual keyboard, however it apparently won't be ready for the initial launch of the device.

Siri can also be used to enter text via dictation.

An App Store will also be offered, complete with third-party software that could be installed, much like the iOS variant. The famous home screen will also make an appearance, compete with a grid of app icons and pinnable widgets. App Store

Powered by a variation of the M2 chip and a dedicated graphics processor called the "Reality Processor," the headset will have two ultra-resolution displays developed with Sony, as well as a collection of cameras to handle pass-through mode. To ensure the screen is visible to glasses users, Apple will offer custom lenses that can rest inside the enclosure itself.

Power is derived from a battery pack, which will be approximately the size of two iPhone 14 Pro Maxes stacked on each other, with power lasting for up to two hours.

However, a prototype headset has the battery pack designed to fit into a user's pocket, rather than being attached to the headset's headband or incorporated into the headset itself. This tethered approach may cause problems for users in the long run, but it will also reduce the weight of the headset sat on their heads.

Current speculation is for a launch unveiling the headset in early 2023, for release later in the year for around $3,000.

As part of Apple's promotional efforts, it may go down the route of the Apple Watch in creating a store-within-a-store, so customers can try out the headsets before buying it.

This could provide Apple with a byproduct of extra sales, as a customer may end up acquiring an iPad or other products from the store, even if they don't pay for the headset itself.

Read on AppleInsider

Comments

  • Reply 1 of 16
    dk49dk49 Posts: 267member
    Just like how I imagined it. This will be a major product coming from Apple after the iPhone. Will create a new platform just like the iPhone which will allow whole companies and ecosystems to exist on top of it. Though of course it will take time to get it mass market because of high price and bulk in the initial versions.
    InspiredCodewatto_cobra
  • Reply 2 of 16
    entropysentropys Posts: 4,152member
    Maybe DK, we will have to see. As I recall the iPhone was probably the most transformative device since both the PC and New Wave CDs that absolutely everyone could see the benefit of owning, and the iPhone was $500.

    The Achilles heels to this proposal is the $3000 price tag, and that it isn’t something you take with you on the go.
    for wide uptake, what you can do with it would have to be incredibly life changing for users. And arguably essential.
    Despite being a headset that covers the head, FaceTime will still be usable, with the headset realistically rendering a user's face and body in VR as an avatar. This would be wildly different from Meta's approach, which relies on cartoon-like avatars instead. 
    Oh, God, a virtual world full of Max Headrooms.
    Alex1NFileMakerFellerh4y3smuthuk_vanalingammobirdwatto_cobra
  • Reply 3 of 16
    thttht Posts: 5,421member
    To navigate the environment, hand, finger, and eye tracking will be used via several cameras and sensors, with gestures such as pinching their thumb and index finger used to trigger items. Eye tracking will allow users to select elements from a menu, without clumsily doing so with their hands. 
    ...
    To enter text into the system, users will still have an option to do it entirely in-headset. Technology is being made to allow users to type in midair with their hands on a virtual keyboard, however it apparently won't be ready for the initial launch of the device.
    This better not be like Minority Report. My arms already feel tired. Not to mention reality has already copied all the policing, privacy and advertising insanity from the movie.


    Power is derived from a battery pack, which will be approximately the size of two iPhone 14 Pro Maxes stacked on each other, with power lasting for up to two hours. ... However, a prototype headset has the battery pack designed to fit into a user's pocket, rather than being attached to the headset's headband or incorporated into the headset itself.
    I find this really odd! They made the decision to have an external battery pack, but they design a battery pack that can only power the device for 2 hours. That's nuts? Weird. Well, surely that means there will be 3rd party packs.

    One thing to note is that I find it implausible that they could have an M1 Max like SoC embedded in the goggles. That's a huge heat source. Plain ski goggles make people sweat. Add 50 W to it? No way. So perhaps the computing bits actually in the "battery pack" too. Once that happens, the whole deal of having a pocket sized battery pack only powering it for two hours makes a little more sense.


    The operating system, xrOS, will be completely in 3D
    I can already feel myself feeling nauseous. rOS is better than xrOS.

    FileMakerFellerwatto_cobra
  • Reply 4 of 16
    blastdoorblastdoor Posts: 3,258member
    This is a great example of why it's good to own your own retail stores. I bet hordes of people will visit Apple Stores to try these things out. 
    InspiredCodeFileMakerFellerwatto_cobra
  • Reply 5 of 16
    blastdoorblastdoor Posts: 3,258member
    tht said:
    Power is derived from a battery pack, which will be approximately the size of two iPhone 14 Pro Maxes stacked on each other, with power lasting for up to two hours. ... However, a prototype headset has the battery pack designed to fit into a user's pocket, rather than being attached to the headset's headband or incorporated into the headset itself.
    I find this really odd! They made the decision to have an external battery pack, but they design a battery pack that can only power the device for 2 hours. That's nuts? Weird. Well, surely that means there will be 3rd party packs.

    One thing to note is that I find it implausible that they could have an M1 Max like SoC embedded in the goggles. That's a huge heat source. Plain ski goggles make people sweat. Add 50 W to it? No way. So perhaps the computing bits actually in the "battery pack" too. Once that happens, the whole deal of having a pocket sized battery pack only powering it for two hours makes a little more sense.

    Regarding heat -- it depends on what voltage and clock speed the thing runs at. Maybe this thing is running under 2 GHz, not over 3 as in a Mac. 
    watto_cobra
  • Reply 6 of 16
    lkrupplkrupp Posts: 10,557member
    dk49 said:
    Just like how I imagined it. This will be a major product coming from Apple after the iPhone. Will create a new platform just like the iPhone which will allow whole companies and ecosystems to exist on top of it. Though of course it will take time to get it mass market because of high price and bulk in the initial versions.
    No way José. While this might be a nerd wet dream come true the mass market will reject it out of hand. These things will never be accepted being worn in public. Remember Glassholes. Today’s commercials still make fun of people wearing such devices in public flailing their hands in the air. And no amount of marketing will force this down the public’s throat. Will it have use in an educational setting, an engineering setting, a architectural or creative setting? Absolutely but at the speculated price it will not be the next big thing.
    muthuk_vanalingamharrykatsaroswatto_cobra
  • Reply 7 of 16
    entropysentropys Posts: 4,152member
    I wouldn’t mind if this thing was desired in the engineering field and got CAD software like Siemens Nx and SolidWorks to start making decent Mac/VR version of their software. Then I could get a MBP and not have to buy a battery life sucking thinkpad.
    watto_cobra
  • Reply 8 of 16
    mattinozmattinoz Posts: 2,299member
    Hasn't MacOS/iOS been in 3d per say for a very long time?
    I mean the display composition engine has been a 3D openGL (assume now Metal) scene since Leopard or slightly before. Allowing all the flying window and stacking effects driven by the OS. Apps stream textures to the scene of what is visible in their windows. 

    Seem like xrOS is just the next step with 2 cameras/eyes looking at the scene to set the depth location of the interface elements.

    To me the question is can 3D content apps stream a model to the viewport of the app window so you can a stereo view of the model when connected to macOS apps.
    watto_cobra
  • Reply 9 of 16
    harrykatsarosharrykatsaros Posts: 75unconfirmed, member
    I hope this thing is great for movie watching. If I can get an IMAX level big screen experience from a set of goggles, I may never leave the house again LOL. 
    watto_cobra
  • Reply 10 of 16
    harrykatsarosharrykatsaros Posts: 75unconfirmed, member
    lkrupp said:
    dk49 said:
    Just like how I imagined it. This will be a major product coming from Apple after the iPhone. Will create a new platform just like the iPhone which will allow whole companies and ecosystems to exist on top of it. Though of course it will take time to get it mass market because of high price and bulk in the initial versions.
    No way José. While this might be a nerd wet dream come true the mass market will reject it out of hand. These things will never be accepted being worn in public. Remember Glassholes. Today’s commercials still make fun of people wearing such devices in public flailing their hands in the air. And no amount of marketing will force this down the public’s throat. Will it have use in an educational setting, an engineering setting, a architectural or creative setting? Absolutely but at the speculated price it will not be the next big thing.
    I completely agree but, outside of the specialist cases that you mentioned, I do think they could become mass market commodity devices if they focus on gaming and movies. You don’t walk around in public with your TV or PlayStation but everybody has one and uses it regularly. imagine going home, putting on a pair of goggles and all of sudden it’s like you’ve got an enormous big screen cinema in your living room while you’re lying on the couch. 
    edited January 2023 muthuk_vanalingamdk49watto_cobra
  • Reply 11 of 16
    dk49dk49 Posts: 267member
    lkrupp said:
    dk49 said:
    Just like how I imagined it. This will be a major product coming from Apple after the iPhone. Will create a new platform just like the iPhone which will allow whole companies and ecosystems to exist on top of it. Though of course it will take time to get it mass market because of high price and bulk in the initial versions.
    No way José. While this might be a nerd wet dream come true the mass market will reject it out of hand. These things will never be accepted being worn in public. Remember Glassholes. Today’s commercials still make fun of people wearing such devices in public flailing their hands in the air. And no amount of marketing will force this down the public’s throat. Will it have use in an educational setting, an engineering setting, a architectural or creative setting? Absolutely but at the speculated price it will not be the next big thing.
    I agree that initially it will be more of a niche device. But that's why I said it will take time. This will set the stage for true AR glasses, which definitely have the potential to become mass market and be the next iPhone. And I am sure Apple will do a much better job than Google's Glassholes. Apple is the last company to churn out something that's not aesthetically pleasing. 
    edited January 2023 watto_cobra
  • Reply 12 of 16
    blastdoorblastdoor Posts: 3,258member
    lkrupp said:
    dk49 said:
    Just like how I imagined it. This will be a major product coming from Apple after the iPhone. Will create a new platform just like the iPhone which will allow whole companies and ecosystems to exist on top of it. Though of course it will take time to get it mass market because of high price and bulk in the initial versions.
    No way José. While this might be a nerd wet dream come true the mass market will reject it out of hand. These things will never be accepted being worn in public. Remember Glassholes. Today’s commercials still make fun of people wearing such devices in public flailing their hands in the air. And no amount of marketing will force this down the public’s throat. Will it have use in an educational setting, an engineering setting, a architectural or creative setting? Absolutely but at the speculated price it will not be the next big thing.
    I completely agree but, outside of the specialist cases that you mentioned, I do think they could become mass market commodity devices if they focus on gaming and movies. You don’t walk around in public with your TV or PlayStation but everybody has one and uses it regularly. imagine going home, putting on a pair of goggles and all of sudden it’s like you’ve got an enormous big screen cinema in your living room while you’re lying on the couch. 
    Also, as a passenger on a plane, train, or automobile it could be nice to have a huge virtual screen for entertainment and work.

    one thing that strikes me as funny is the bit about getting to use your actual face in FaceTime calls —guess what, I already get to do that! In fact I’ve been on Zoom calls with dozens of people and their actual faces. So I’m not sure how much of a killer feature that is.
  • Reply 13 of 16
    williamhwilliamh Posts: 1,032member
    lkrupp said:
    . Will it have use in an educational setting, an engineering setting, a architectural or creative setting? Absolutely but at the speculated price it will not be the next big thing.

    The rumored $3,000 price is just an Apple marketing mind game like the rumors before the iPad was released,  When this device comes at out $1,500, the tech press will marvel at the bargain and more people will be able to afford it.  Your overall premise is, in my view, correct, that it won't be the next big thing at least until a cheaper more mass-market version is released. (Maybe it will be a subscription device?)
    watto_cobra
  • Reply 14 of 16
    badmonkbadmonk Posts: 1,285member
    Love the article and comments.  I was skeptical about the speed of adoption and I think it will be like the Apple Watch…slow at first but the. will snowball.  I think about the screen dependency of Generation Z and how they have adjusted to having compact simpler somewhat disconnected lives.  If the device substitutes for a home theater or even large display they will buy.

    Also Apple is so uniquely positioned to pull this off…hardware/software integration, computing efficiency (huge here for weight and heat issues), plus its primesense acquisition and ability to integrate real world motion as a simulacra.

    Plus I will say it again, spectator sports will be the thing that will be unexpected boon.

    Afterall if people can put this on their head, its not a stretch to wear VR wear.  And imagine how great it would be to be immersed in game action?


    watto_cobra
  • Reply 15 of 16
    MacProMacPro Posts: 19,718member
    entropys said:
    Maybe DK, we will have to see. As I recall the iPhone was probably the most transformative device since both the PC and New Wave CDs that absolutely everyone could see the benefit of owning, and the iPhone was $500.

    The Achilles heels to this proposal is the $3000 price tag, and that it isn’t something you take with you on the go.
    for wide uptake, what you can do with it would have to be incredibly life changing for users. And arguably essential.
    Despite being a headset that covers the head, FaceTime will still be usable, with the headset realistically rendering a user's face and body in VR as an avatar. This would be wildly different from Meta's approach, which relies on cartoon-like avatars instead. 
    Oh, God, a virtual world full of Max Headrooms.
    Maybe Apple will push the Apple Card purchase, only $100 a month interest free.
  • Reply 16 of 16
    blastdoor said:
    lkrupp said:
    dk49 said:
    Just like how I imagined it. This will be a major product coming from Apple after the iPhone. Will create a new platform just like the iPhone which will allow whole companies and ecosystems to exist on top of it. Though of course it will take time to get it mass market because of high price and bulk in the initial versions.
    No way José. While this might be a nerd wet dream come true the mass market will reject it out of hand. These things will never be accepted being worn in public. Remember Glassholes. Today’s commercials still make fun of people wearing such devices in public flailing their hands in the air. And no amount of marketing will force this down the public’s throat. Will it have use in an educational setting, an engineering setting, a architectural or creative setting? Absolutely but at the speculated price it will not be the next big thing.
    I completely agree but, outside of the specialist cases that you mentioned, I do think they could become mass market commodity devices if they focus on gaming and movies. You don’t walk around in public with your TV or PlayStation but everybody has one and uses it regularly. imagine going home, putting on a pair of goggles and all of sudden it’s like you’ve got an enormous big screen cinema in your living room while you’re lying on the couch. 
    Also, as a passenger on a plane, train, or automobile it could be nice to have a huge virtual screen for entertainment and work.

    one thing that strikes me as funny is the bit about getting to use your actual face in FaceTime calls —guess what, I already get to do that! In fact I’ve been on Zoom calls with dozens of people and their actual faces. So I’m not sure how much of a killer feature that is.
    Because you could be standing anywhere in the world, maybe exploring a city in Apple Maps together?
Sign In or Register to comment.