Apple's headset faced numerous snags early, Jony Ive still involved with project

Posted:
in General Discussion
A new report details more of the hurdles that Apple has faced in developing its mixed-reality headset, including one stemming from a decision by former Apple design chief Jony Ive.

Apple headset render
Apple headset render


According to The Information, Apple made a decision in 2019 to continue work on a standalone headset with its own processing power instead of one paired with a base station. That's despite the fact that the base station unit had superior graphics and photorealistic avatars.

Mike Rockwell, the executive in charge of Apple's VR team, believed the base station model would win out among Apple brass because he thought they wouldn't accept the cartoony graphics of the standalone model. However, he was wrong.

Ive pushed for the standalone version since the earliest days of development, and Apple's top executives sided with him. That choice has reportedly had lasting repercussions.

In the years since that decision was made in 2019, Apple's team working on the project would struggle to overcome technical hurdles related to chips, cameras, and avatars. Those hurdles have had real effects. A repot from January indicated that Apple cancelled plans to debut the device at WWDC 2022.

People familiar with the matter say that Rockwell has failed to deliver the premium MR experience he promised Apple executives. Apple executives are expecting an AR experience beyond rivals like Meta, with virtually no latency and advanced graphics and body tracking.

Some executives blame Ive, who The Information says fundamentally changed the product's main goal. Originally, it was meant to be a device creatives and professionals would use at a desk. Ive envisioned it to be a portable product.

Advocates for the desk bound version said that Apple should have released a device that creatives could use to create mixed-reality experiences before moving onto something for consumers, such as "Apple Glass."

Other details in the report include the fact that Apple tapped Kim Vorrath to help discipline the team, which had previously acted more like a "freewheeling" startup within Apple. The report also claims that Apple's team took MR units disguised as surveying equipment into the field to test digital map creation.

One app that Rockwell and his team wanted to wow executives with was a FaceTime-like system with photorealistic avatars that could make participants feel like they were in the same location. Apparently, one attempt at the app backfired, since the photorealistic avatars had crossed into "uncanny valley" territory.

Ive's influence still lingers over the project. Although the Apple design chief left the company in 2019, he still reportedly consults with the iPhone maker on the headset. He reportedly continues to tweak the design of the device.

Apple is continuing to work toward reducing component costs for the device, and is also working with other companies like Unity Technologies to allow third-party developers to create software for the headset.

Read on AppleInsider
«1

Comments

  • Reply 1 of 37
    mike1mike1 Posts: 2,981member
    Is anyone really surprised that...

    1. Apple decided to push for the best user experience and make a device that wasn't tethered to another? There is absolutely no need to or argument for tethering to a phone. Whatever 'A" processor is deemed necessary from the phone can be incorporated into a standalone device.

    2. That Apple decided to pursue a product for customers beyond "creatives". That would be like limiting Apple Watch customers to time keepers at a football or hockey game.

    3. That they wouldn't try to leapfrog the competition from that user experience point of view.

    4. It wasn't easy or quick or cheap to do that.
    jas99Alex1Nmacguilolliverwatto_cobra
  • Reply 2 of 37
    tjwolftjwolf Posts: 424member
    mike1 said:
    Is anyone really surprised that...

    1. Apple decided to push for the best user experience and make a device that wasn't tethered to another? There is absolutely no need to or argument for tethering to a phone. Whatever 'A" processor is deemed necessary from the phone can be incorporated into a standalone device.

    2. That Apple decided to pursue a product for customers beyond "creatives". That would be like limiting Apple Watch customers to time keepers at a football or hockey game.

    3. That they wouldn't try to leapfrog the competition from that user experience point of view.

    4. It wasn't easy or quick or cheap to do that.
    For (1), I don't know if anyone was talking about a physical tether.  The argument for a wireless tethering to a phone is obvious:  the weight you're willing to put up with on your face is substantially less than what you're willing to put up with in a phone that's in your hand or in your pocket most of the time.

    For (2), I agree that it shouldn't be a surprise that Apple needs to address a larger audience than that of creatives at their desks.  But even an audience as large as "serious gamers" would really be too small for Apple.  But those are the only ones who'd put up with a heavy VR headset.  For Apple to sell millions of devices, they need something as light as glasses which people are willing to wear for extended periods - i.e. a whole day - which means the vaunted AR glasses.

    Totally agree with (3) - but with the product they SHOULD be aiming for, there is currently no competition anyway.  Nobody's come out with a lightweight, useful AR/MR headset/glasses that people are actually buying in any volume.

    Yes, designing headwater that hundreds of millions of people would be willing to buy is definitely hard.  I always thought (and still think should) Apple should focus on a stylish pair of AR glasses that use the iPhone for all the heavy computational lifting as that's really the only way to get a whole day of wear out of something that has so little room for batteries.
    williamhAlex1Nwatto_cobra
  • Reply 3 of 37
    thrangthrang Posts: 926member
    I own everything Apple, but I have a tough time with the use case for this. I'm not a gamer, but I would think from Apple's perspective, a gamer focus "mostly" would be a long-term failure.

    I've thought AR for vertical business uses would be a great start - one example is the financial industry, where most everyone is looking at 2-4 large monitors. Imagine the space and cost savings (though would need to be 8k resolution minimum to start effectively replacing the real estate of multiple monitors I suppose). Medical, technical documentation, legal, security, etc. Many verticals, but will depend on strong developer support...


    designrAlex1Nwatto_cobra
  • Reply 4 of 37
    mike1mike1 Posts: 2,981member
    thrang said:
    I own everything Apple, but I have a tough time with the use case for this. I'm not a gamer, but I would think from Apple's perspective, a gamer focus "mostly" would be a long-term failure.

    I've thought AR for vertical business uses would be a great start - one example is the financial industry, where most everyone is looking at 2-4 large monitors. Imagine the space and cost savings (though would need to be 8k resolution minimum to start effectively replacing the real estate of multiple monitors I suppose). Medical, technical documentation, legal, security, etc. Many verticals, but will depend on strong developer support...



    Apple has a knack for creating the use case that we haven't thought of yet. Targeting serious gaming will probably not result in any significant sales. However, maybe it becomes the decade's GameBoy for casual gamers. Maybe they will show a compelling app/capability/use case that nobody considered before. The Apple Watch didn't come to be because people needed a new way to tell time. Sales really started to take off when they targeted and created hardware and apps to support fitness. That plus the existing communication/connectivity features was compelling to people who didn't typically wear watches or might have settled for some type of Fitbit device.
    jas99williamhJapheyAlex1NJaiOh81fastasleeplolliverwatto_cobra
  • Reply 5 of 37
    AniMillAniMill Posts: 64member
    I’m of the mindset that anything tech that places an undue burden upon the user is always doomed to fail mass-market. Current AR/VR is too niche, and annoying to wear. And there is one factor that is always overlooked: our eyes expect to do both the parallax and focus adjustments concurrently, but because VR screens are locked to the set distance-even with optical correction-our minds want the refocus as we change parallax. This is one cause of headaches (I experience this also with 3D movies, which also failed because of the requirement for users to wear 3D glasses).

    I’m sure Apple’s entry will be extraordinary and cool, but it’ll be prohibitively expensive, yet still have the same focus/parallax limitations of every AR/VR headset.
    designrwatto_cobra
  • Reply 6 of 37
    dk49dk49 Posts: 208member
    The Apple Watch also initially relied on the iPhone. So if the AR/VR headset relies on the iPhone in the beginning, it's fine. Atleast it will be lighter and cheaper, allowing more people to buy.
    jas99Alex1Nwatto_cobra
  • Reply 7 of 37
    MarvinMarvin Posts: 14,753moderator
    People familiar with the matter say that Rockwell has failed to deliver the premium MR experience he promised Apple executives. Apple executives are expecting an AR experience beyond rivals like Meta, with virtually no latency and advanced graphics and body tracking.

    Some executives blame Ive, who The Information says fundamentally changed the product's main goal. Originally, it was meant to be a device creatives and professionals would use at a desk. Ive envisioned it to be a portable product.
    The portable one makes more sense because it can work for both mobile and static environments. More advanced imagery can be streamed from a laptop/desktop. This is what the nreal Air glasses do:



    This has a cable tethered but it can work wirelessly. When it's processing internally, it can render basic UI and stream movies. When it is rendering realistic 3D objects, it can stream from an iOS device or Mac.

    Passthrough cameras would give better quality images because it can make the digital overlays fully opaque.
    Alex1Nwatto_cobra
  • Reply 8 of 37
    designrdesignr Posts: 700member
    While I don't doubt the rumors about this product. I still can't figure this out for Apple.

    Apple is—I assume—looking to move tens of millions of units (or at least millions). It's unclear to me how and why this product does that. Phones? Duh. Watches. Yep. Tablets. Sure. Heck, I can even imagine doing something like the Oura ring at some point. But this type of device seems to lack a compelling use case for millions or tens of millions of people.

    This feels like a "others are doing this product and we can do it really well" play without consideration of the real problem they're trying to solve or the need they're trying to meet.

    Aside from gaming—which might be a viable multi-million unit market—what's the large (and broad) scale use case?
    edited May 20 chadbagAlex1Nwatto_cobra
  • Reply 9 of 37
    chadbagchadbag Posts: 1,669member
    Apple has a conundrum :   they have so much money that they need outlets for that they are starting to throw crap on the wall to see what sticks, but to make it worthwhile, they have to sell so many of them that they have to realky identify (or create) the new hit thing and hit a home run with it, or drop it (at a level that would be a success for a smaller company).  

    I don't see how this (or the rumored Apple Car) fit the Apple need.  Easy to throw money at but hard to see where it makes enough money for them to be meaningful.  

    I for one have no interest in AR or VR fo anything other than the novelty of it.   I wore gasses for almost 30 years -- no desire to go back to that.  I'll wear sunglasses when necessary but remove them as soon as I can. Sweaty and dirty lenses in front of my eyes -- no thanks.  

    CarPlay makes sense since it helps sell more phones and locks you into the Eco system more.  Car is low margin, intensive  capital commitment,  comparatively, and makes no sense. 

    Same with VR-AR. Iphone and watch make sense because I choose when to look at them and can avoid them if I need a digital break.  AR/VR are a more full time commitment (harder to store when not in use and harder to use when not worn all the time).  So it becomes more a full time digital intrusion into your life with little benefit. 

    Vertical market use makes sense for AR/VR but can that bring the volume of sales and success Apple needs for a company their size.  
    designrAlex1Nopinionanantksundaramwatto_cobra
  • Reply 10 of 37
    designrdesignr Posts: 700member
    chadbag said:
    Apple has a conundrum :   they have so much money that they need outlets for that they are starting to throw crap on the wall to see what sticks, but to make it worthwhile, they have to sell so many of them that they have to realky identify (or create) the new hit thing and hit a home run with it, or drop it (at a level that would be a success for a smaller company).  

    I don't see how this (or the rumored Apple Car) fit the Apple need.  Easy to throw money at but hard to see where it makes enough money for them to be meaningful.  

    I for one have no interest in AR or VR fo anything other than the novelty of it.   I wore gasses for almost 30 years -- no desire to go back to that.  I'll wear sunglasses when necessary but remove them as soon as I can. Sweaty and dirty lenses in front of my eyes -- no thanks.  

    CarPlay makes sense since it helps sell more phones and locks you into the Eco system more.  Car is low margin, intensive  capital commitment,  comparatively, and makes no sense. 

    Same with VR-AR. Iphone and watch make sense because I choose when to look at them and can avoid them if I need a digital break.  AR/VR are a more full time commitment (harder to store when not in use and harder to use when not worn all the time).  So it becomes more a full time digital intrusion into your life with little benefit. 

    Vertical market use makes sense for AR/VR but can that bring the volume of sales and success Apple needs for a company their size.  
    All of this. Very well put. My only disagreement—weakly held—is about the car. I could see Apple finding some way to "make the whole widget" here in a profitable way that does (in some way) to our car experience what they did with phones. But it's a long shot to be sure. Making a car is vastly more complicated than making a phone (or watch, or tablet, or computer).

    I think Apple is trying (perhaps somewhat late and somewhat desperately) to address their conundrum with the shift to service/subscription/drip revenue and lock in their ecosystem for the really long term. And it might work. Still too early to tell. I'm skeptical here too.

    Apple's core competency—what they, and they alone, are amazing at—is the almost magical marriage of digital hardware and software. It's not that they cannot expand into other areas. It's just quite hard. They could be big enough that just becoming a conglomerate is the way to go, but history is not kind to most of those. And it doesn't really fit the Apple "vibe" or "ethos".

    Perhaps, the better strategic question for them is where else can this skill set be applied to serve millions of customers? What other devices in our lives would benefit from the "Apple touch"? TVs to come to mind (or as someone in another thread suggested a HomePod + Apple TV soundbar that gets you 99% of the way there without shipping TVs...kinda like CarPlay...all the Apple without the risk of doing the whole thing). Home gear (a la Nest, etc.) is another way to go. Certainly this AR/VR/MR headset is such an attempt. I just don't see a broad market for it.

    Alex1Nwatto_cobra
  • Reply 11 of 37
    "Originally, it was meant to be a device creatives and professionals would use at a desk."

    What is this, 1999?
    designrAlex1Nwatto_cobra
  • Reply 12 of 37
    JapheyJaphey Posts: 1,373member
    All these AR/VR haters will probably all be online or in line bright and early on launch day. 
    watto_cobra
  • Reply 13 of 37
    larryjwlarryjw Posts: 917member
    I don't understand the argument that Apple must push out the ultimate device to millions before putting out anything.

    Just getting anything (of quality) out the door is a benefit. Apple certainly would want to push out a product superior to the competition, otherwise why bother. 

    So, I might guess that Apple has determined they can't make the product, better than the competition, except by making the product standalone.
    watto_cobra
  • Reply 14 of 37
    designrdesignr Posts: 700member
    Japhey said:
    All these AR/VR haters will probably all be online or in line bright and early on launch day. 
    Who are the "haters"?
    watto_cobra
  • Reply 15 of 37
    designrdesignr Posts: 700member
    larryjw said:
    I don't understand the argument that Apple must push out the ultimate device to millions before putting out anything.

    Just getting anything (of quality) out the door is a benefit. Apple certainly would want to push out a product superior to the competition, otherwise why bother. 

    So, I might guess that Apple has determined they can't make the product, better than the competition, except by making the product standalone.
    I'm not arguing that they must initially sell millions. But I do think that Apple's general outlook—and need—is to have something where there will be a large market. Some of this is a function of being a large publicly-traded company that must look for new (and large) growth opportunities.


    edited May 21 watto_cobra
  • Reply 16 of 37
    eightzeroeightzero Posts: 2,689member
    The only consumer case for this, other than the already discussed ones, that I can come up with is some sort of remote in-store shopping device. You pop these on and you're walking the aisles of a store to pull items off a shelf, compare prices, check inventory. But crikey...thats a huge stretch.

    Other commentors hit have the issue pretty well. I don't get what this device is for, other than for gamers. And I don't seem Apple being committed to the gaming market like the others - Nintendo, playstation - are. I'm not a gamer, so I'd need to defer to those that know it better. Is there really a market for Apple to make this a game changer? 
    watto_cobra
  • Reply 17 of 37
    MarvinMarvin Posts: 14,753moderator
    designr said:
    Aside from gaming—which might be a viable multi-million unit market—what's the large (and broad) scale use case?
    Movies. It gives you a virtual display as big as 200" anywhere you are and in true 3D and in some cases OLED.

    Art. It gives you an infinite canvas for drawing onto arbitrary surface anywhere you are, also in 3D.

    Music. It lets you attend virtual concerts and watch a live stage from the sofa.

    Sports. You can pay for a virtual camera sitting at a live sporting event and feel like you are there.

    Gaming is a big use case for this because it can be made to work with any gaming device. People can connect a Nintendo Switch to it and play with a big virtual screen anywhere.

    Education. Currently educators and parents are competing with devices for a child's attention and failing. They are losing to gaming. AR/VR devices can gamify education.





    What holds VR back is most of the hardware is too expensive (Oculus Quest is one of the few at a mainstream price), too bulky and they can't be used much for the above due to the form factor and technical setup they need. Apple's hardware is always done in a way that is intuitive to use. There were tablets before the iPad but they were only used in business. When iPads arrived, kids could use them easily.

    With a mass-market price point (starting under $799), lightweight form factor, good first party software support (app store + easy to access movie content), they can easily sell millions of units per year.
    eightzerowatto_cobra
  • Reply 18 of 37
    designrdesignr Posts: 700member
    Marvin said:
    designr said:
    Aside from gaming—which might be a viable multi-million unit market—what's the large (and broad) scale use case?
    Movies. It gives you a virtual display as big as 200" anywhere you are and in true 3D and in some cases OLED.

    Art. It gives you an infinite canvas for drawing onto arbitrary surface anywhere you are, also in 3D.

    Music. It lets you attend virtual concerts and watch a live stage from the sofa.

    Sports. You can pay for a virtual camera sitting at a live sporting event and feel like you are there.

    Gaming is a big use case for this because it can be made to work with any gaming device. People can connect a Nintendo Switch to it and play with a big virtual screen anywhere.

    Education. Currently educators and parents are competing with devices for a child's attention and failing. They are losing to gaming. AR/VR devices can gamify education.





    What holds VR back is most of the hardware is too expensive (Oculus Quest is one of the few at a mainstream price), too bulky and they can't be used much for the above due to the form factor and technical setup they need. Apple's hardware is always done in a way that is intuitive to use. There were tablets before the iPad but they were only used in business. When iPads arrived, kids could use them easily.

    With a mass-market price point (starting under $799), lightweight form factor, good first party software support (app store + easy to access movie content), they can easily sell millions of units per year.
    Thanks. Interesting ideas. I'm still skeptical.

    Movies, Music, Sports: So I need to buy a headset ($800 a pop) for everyone in my family to enjoy a movie (basically now in isolation) when we could just sit back on the couch and enjoy some snacks and watch a large flat screen together? This seems like more work (and cost) than necessary for the additional benefits.

    Art, Education: Um, okay. Sure. I'm old enough now to just chuckle at suggestions that some new amazing technology will be used to revolutionize things like art or education. TV was going to be that. LOL. I'm sure there are applications here. But I'm skeptical this moves lots of units.

    Gaming: Yep. I get it. This is the one that does make the most sense. In part because a lot of gaming actively strives to create immersive experiences. I could even see some physical, location-based games like escape rooms being done in this way. Maybe. But even escape rooms have an interactive social aspect to not be discounted.

    And at the bottom of it all, these examples end up doing things that isolating people from the social experiences of enjoying these things (e.g., watching movies, concerts, or sporting events.) I'm curious whether mass numbers of people actually want to do that on any kind of regular or consistent basis...enough to justify the cost.

    And any argument that suggests "this can/will be social" and "you can/will be interacting with others" ignores the neurological and physiological aspects of real (face-to-face, in-person, physical proximate) social experiences. This is a very human thing. You could say it's one of the things that makes us human. I'm not sure people are keen to give that up. Oh sure, they will try it. They'll experiment. They'll dip their toe in the water. But I think they will end up finding it wanting/lacking something that can be achieved without an expensive device strapped to their face.

    TBH this feels a lot like Dean Kamen's aspirational ideas of complete cities being built (or re-configured) around the Segway personal transporter. It is a great technology. But it just didn't shake out as a change the world kind of thing and ended up being just a niche product.

    So...I may end up being completely wrong. But I'm definitely a skeptic at this point.
    edited May 21 watto_cobra
  • Reply 19 of 37
    thttht Posts: 4,388member
    With the recent round of media for the After Steve book, more rumors, et al, it really does seem that Apple needs a product visionary, or a Chief Product Officer, someone who can focus all these product ideas into a buildable and shippable form, then iterate. And, that person can't be Jony Ive, which Cook initially thought he would be. Feels like I'm taking crazy pills as Cook essentially gave Ive an annual billion dollar budget to play around with car ideas and other product ideas, only as a way to keep him at Apple and there wasn't really an expectation that he'd produce, otherwise his budget would have been cut to zero. He should have let him go in 2015 like he wanted to after the Apple Watch was done.

    This VR goggles thing is also crazy. How is it that Rockwell was building something the executive team didn't want? They just let him do whatever he wanted and checked in on him every once in a while? His team is spending billions. I do agree that Rockwell was building the wrong product, and Ive's suggestion of an independent mobile product was wrong too.

    I was envisioning AR glasses, not goggles. It wasn't going to be smart. It was going to be a high resolution projector, one for each eye, that would create virtual objects in front of your eyes. It would have the full suite of sensors: cameras, LIDAR, IR, GPS, magnetometer, IMU, temperature, etc. Enough battery for a day, and charge over night. And processing was all going to be done on an iPhone. Essentially the glasses would be an AirPlay target for your iPhone, across a low latency wireless protocol. Undecided on whether it would have speakers or headphone attachment.

    You use it to augment your vision. You walk around, drive around, it would provide information like street names, businesses etc.. There would be navigation instructions, time, temperature, weather prediction. You could use it as a camera, for both videos and stills, with near-time language translation. You use it at a desk, and it would project a virtual screen, fixed next to your laptop or desk display, and would display stuff, like videos, music, etc. People could buy multiple pairs, one for home, one for the car, etc.

    Rockwell's idea of stationary VR goggles seems crazy. Apple doesn't have a platform of software for it, which is mostly 3D games. People aren't going to be wearing goggles all day. After about 7 years of VR, it doesn't seem to be much more than platform for 3D games. Ive's idea of it being independent with a builtin SoC is about 10 years into the future before an SoC lower power enough would fit. Either Joswiak, as the chief product marketing officer, needs to step in and focus these ideas, or Cook needs to oversee this more and delegate the gov't interaction stuff to the Chief Counsel.
    anantksundaram
  • Reply 20 of 37
    MarvinMarvin Posts: 14,753moderator
    designr said:
    Movies, Music, Sports: So I need to buy a headset ($800 a pop) for everyone in my family to enjoy a movie (basically now in isolation) when we could just sit back on the couch and enjoy some snacks and watch a large flat screen together? This seems like more work (and cost) than necessary for the additional benefits.
    The price would definitely hinder uptake if it's beyond a certain level. Ideally a viewer device that doesn't have heavy on-board processing would have around a $499 starting price. The Oculus Quest 2 starts at $299 so $499 should be doable to make something that rivals this. If they sell them with iPhones, they can bundle them with the monthly payments. Families buy expensive iPhones for family members because they pay monthly.

    I imagine that they won't have a new audio system but will rely on existing audio hardware like AirPods Pro. If they had to bundle audio hardware that would add to the price. They could always bundle some basic buds.
    designr said:
    Art, Education: Um, okay. Sure. I'm old enough now to just chuckle at suggestions that some new amazing technology will be used to revolutionize things like art or education. TV was going to be that. LOL. I'm sure there are applications here. But I'm skeptical this moves lots of units.
    The iPad did this already. The iPad + Procreate + Apple Pencil is used a lot for making art now. Procreate reportedly has over 3 million users. Adobe has over 10 million.

    It doesn't have to replace the other tools, it would be an extra tool. When someone wants a big canvas to draw on, they can just sit at a large board and draw digitally.

    They'd be ideal in education and businesses because they are compact and could work easily for typing. Just connect up a bluetooth keyboard/mouse and people can type up documents.
    designr said:
    And at the bottom of it all, these examples end up doing things that end up isolating people from the social experiences of enjoying these things (e.g., watching movies, concerts, or sporting events.) I'm curious whether mass numbers of people actually want to do that on any kind of regular or consistent basis...enough to justify the cost.

    And any argument that suggests "this can/will be social" and "you can/will be interacting with others" ignores the neurological and physiological aspects of real (face-to-face, in-person, physical proximate) social experiences. This is a very human thing. You could say it's one of the things that makes us human. I'm not sure people are keen to give that up. Oh sure, they will try it. They'll experiment. They'll dip their toe in the water. But I think they will end up finding it wanting/lacking something that can be achieved without an expensive device strapped to their face.
    It wouldn't replace in-person scenarios, it's another way of experiencing content. The Apple Watch doesn't offer much over existing devices but has sold millions of units because it offers some unique features.

    I think once people can put on some glasses that are comfortable to wear and view a huge virtual display, they will sell a few million units. Students would love these for binge-watching TV shows.

    Once there's some hardware available it will be clearer how likely a lot of people will use it but some things are certain. If it's anywhere near the $3k price point like Microsoft's HoloLens, it will be DOA. If it's too bulky like the standard VR hardware, it won't hit mass adoption. For mainstream use and mass adoption, it will have to be both comfortable/lightweight and have a reasonable entry price point.
    edited May 21
Sign In or Register to comment.