Apple AR headset will use Face ID tech for hand gesture tracking, says Kuo

Posted:
in Future Apple Hardware
Analyst Ming-Chi Kuo now claims that the forthcoming Apple AR headset will track hand movements, which will allow for the device to interpret gestures.




In his second Apple AR report in as many days, respected analyst Ming-Chi Kuo is reporting more details of the sensors expected to be in the first of Apple's forthcoming headsets.

"Gesture control and object detection are critical human-machine UI designs of Apple's AR/MR headset," writes Kuo in a note for investors. "Apple's AR/MR headset is equipped with more 3D sensing modules than iPhones."

"We predict that Apple's AR/MR headset will have four sets of 3D sensing (vs. one to two sets for iPhone/high-end smartphones)," he continues. "We predict that the structured light of the AR/MR headset can detect not only the position change of the user or other people's hand and object in front of the user's eyes but also the dynamic detail change of the hand."

Saying that this "dynamic detail change" is similar to how Face ID can detect changes of expression, Kuo claims that capturing "the details of hand movement can provide a more intuitive and vivid human-machine UI."

There is a difference between Face ID and this hand tracking, though.

"Although both adopt structured light, the distance between the hand (user or other people's) and the object detected by the headset device needs to be longer than that seen by iPhone's Face ID," continues Kuo, "so the structured light power consumption of the headset device is higher."

"We predict that the detection distance of Apple's AR/MR headset with structured light is 100-200% farther than the detection distance of the iPhone Face ID," he says. "To increase the field of view (FOV) for gesture detection, we predict that the Apple AR/MR headset will be equipped with three sets of ToFs [Time of Flight] to detect hand movement trajectories with low latency requirements."

Kuo's focus is on the impact these components and requirements will have on companies that his clients may choose to invest in. So he reports that the firms WIN semi and Lumentum are believed to be the main providers for the headset's VCSELs (Vertical Cavity Surface-Emitting Laser) sensors.

However, he also backs up his previous report that Apple ultimately intends the AR headset to replace the iPhone.

"The innovative human-machine UI for headset devices requires the integration of many technologies," he writes. "It will be critical for headsets to replace the existing consumer electronic products with displays in the next ten years."

Read on AppleInsider

Comments

  • Reply 1 of 10
    I love VR/AR headsets but they won't be replacing hand held devices for a long, long time if ever. If they got small and light enough, they could replace your computer screen and iPad but the iPhone is in a different category. I keep mine in my back pocket when I am doing yard work which includes swinging a pickaxe. No headset will replace it as I require some degree of eye protection and wont risk damaging either my eyes or an expensive gadget. This is just one personal example but there are many others. Can you use an AR headset while driving? I am guessing not or at least not until your car drives itself. Let's take a look at the technological roadblocks preventing constant use of an AR headset:
    Currently VR headsets are so big and heavy that they become uncomfortable to wear after an hour or two.
    Even if an AR headset were smaller, it would still need to be powered by batteries. Where do they go? In your pocket with a cord dangling down your neck?
    You can shrink the vision system but what about the rest of the electronics? It all generates heat so how to you dissipate it if the electronics are built into a pair of glasses?
    AR is great but for the best virtual experiences, you need full VR. Will the AR glasses have a full VR mode or will we need another headset for that?
    Many people are very sensitive to differences between motion they see with their eyes and what they feel with their sense of balance. How do you do VR without making people sick?
    Finally the cameras. You can't have AR without some cameras to help with the tracking and scene analysis. People threw a hissy fit over the original Google Glass AR glasses because they had cameras. There was at least one fight over them. Is everyone ready to accept a camera on everyone's face constantly recording and sending the video back to the Apple's servers? What happens when Apple starts scanning those videos for anything illegal to protect kids or something?
    JapheyFileMakerFellerBobby_michaelelijahg
  • Reply 2 of 10
    JapheyJaphey Posts: 1,063member
    I love VR/AR headsets but they won't be replacing hand held devices for a long, long time if ever. If they got small and light enough, they could replace your computer screen and iPad but the iPhone is in a different category. I keep mine in my back pocket when I am doing yard work which includes swinging a pickaxe. No headset will replace it as I require some degree of eye protection and wont risk damaging either my eyes or an expensive gadget. This is just one personal example but there are many others. Can you use an AR headset while driving? I am guessing not or at least not until your car drives itself. Let's take a look at the technological roadblocks preventing constant use of an AR headset:
    Currently VR headsets are so big and heavy that they become uncomfortable to wear after an hour or two.
    Even if an AR headset were smaller, it would still need to be powered by batteries. Where do they go? In your pocket with a cord dangling down your neck?
    You can shrink the vision system but what about the rest of the electronics? It all generates heat so how to you dissipate it if the electronics are built into a pair of glasses?
    AR is great but for the best virtual experiences, you need full VR. Will the AR glasses have a full VR mode or will we need another headset for that?
    Many people are very sensitive to differences between motion they see with their eyes and what they feel with their sense of balance. How do you do VR without making people sick?
    Finally the cameras. You can't have AR without some cameras to help with the tracking and scene analysis. People threw a hissy fit over the original Google Glass AR glasses because they had cameras. There was at least one fight over them. Is everyone ready to accept a camera on everyone's face constantly recording and sending the video back to the Apple's servers? What happens when Apple starts scanning those videos for anything illegal to protect kids or something?
    I agree. The smartwatch hasn’t replaced the smartphone (yet), which hasn’t replaced the laptop (yet), which hasn’t replaced the desktop. Yes, there are a lot of functional overlaps between devices, as there will undoubtedly be with HMD’s as well. But, to suggest that they will replace smartphones in 10 years is completely ridiculous. Or possibly it’s just a disingenuous pump of the companies he’s promoting. Either way, nope. 

    In answer to your question, the full VR experience will probably require a separate headset for the foreseeable future, at least until the social norms have changed enough to accept people wearing them around in public. 
    napoleon_phoneapartOutdoorAppDeveloperFileMakerFellerBobby_michael
  • Reply 3 of 10
    mattinozmattinoz Posts: 1,710member
    I love VR/AR headsets but they won't be replacing hand held devices for a long, long time if ever. If they got small and light enough, they could replace your computer screen and iPad but the iPhone is in a different category. I keep mine in my back pocket when I am doing yard work which includes swinging a pickaxe. No headset will replace it as I require some degree of eye protection and wont risk damaging either my eyes or an expensive gadget. This is just one personal example but there are many others. Can you use an AR headset while driving? I am guessing not or at least not until your car drives itself. Let's take a look at the technological roadblocks preventing constant use of an AR headset:
    Currently VR headsets are so big and heavy that they become uncomfortable to wear after an hour or two.
    Even if an AR headset were smaller, it would still need to be powered by batteries. Where do they go? In your pocket with a cord dangling down your neck?
    You can shrink the vision system but what about the rest of the electronics? It all generates heat so how to you dissipate it if the electronics are built into a pair of glasses?
    AR is great but for the best virtual experiences, you need full VR. Will the AR glasses have a full VR mode or will we need another headset for that?
    Many people are very sensitive to differences between motion they see with their eyes and what they feel with their sense of balance. How do you do VR without making people sick?
    Finally the cameras. You can't have AR without some cameras to help with the tracking and scene analysis. People threw a hissy fit over the original Google Glass AR glasses because they had cameras. There was at least one fight over them. Is everyone ready to accept a camera on everyone's face constantly recording and sending the video back to the Apple's servers? What happens when Apple starts scanning those videos for anything illegal to protect kids or something?
    I could see my 3 key devices morphing to VR headset (replacing Mac) for solo and focused work -> IPad (group work where I need to see faces) -> Watch (on the go com's and updates). 

    But AR / VR seems to me like it would struggle to be for group work or client interaction. I can see a good business for a WeWork type company to have group VR spaces. 
    OutdoorAppDeveloper
  • Reply 4 of 10
    radarthekatradarthekat Posts: 3,425moderator
    I love VR/AR headsets but they won't be replacing hand held devices for a long, long time if ever. If they got small and light enough, they could replace your computer screen and iPad but the iPhone is in a different category. I keep mine in my back pocket when I am doing yard work which includes swinging a pickaxe. No headset will replace it as I require some degree of eye protection and wont risk damaging either my eyes or an expensive gadget. This is just one personal example but there are many others. Can you use an AR headset while driving? I am guessing not or at least not until your car drives itself. Let's take a look at the technological roadblocks preventing constant use of an AR headset:
    Currently VR headsets are so big and heavy that they become uncomfortable to wear after an hour or two.
    Even if an AR headset were smaller, it would still need to be powered by batteries. Where do they go? In your pocket with a cord dangling down your neck?
    You can shrink the vision system but what about the rest of the electronics? It all generates heat so how to you dissipate it if the electronics are built into a pair of glasses?
    AR is great but for the best virtual experiences, you need full VR. Will the AR glasses have a full VR mode or will we need another headset for that?
    Many people are very sensitive to differences between motion they see with their eyes and what they feel with their sense of balance. How do you do VR without making people sick?
    Finally the cameras. You can't have AR without some cameras to help with the tracking and scene analysis. People threw a hissy fit over the original Google Glass AR glasses because they had cameras. There was at least one fight over them. Is everyone ready to accept a camera on everyone's face constantly recording and sending the video back to the Apple's servers? What happens when Apple starts scanning those videos for anything illegal to protect kids or something?
    Add those of us with compromised vision and it might take even longer before we will be able to retire our handsets.  
    napoleon_phoneapartOutdoorAppDevelopermichelb76
  • Reply 5 of 10
    FaceID & detailed face tracking, LiDAR, head tracking (AirPods) & spatial audio, etc. are each individually awesome and valuable, but I’m excited to experience them (and of course more) work together on this! 
    Bobby_michael
  • Reply 6 of 10
    I like how we won't have to deal with clucky controllers no more, and maybe Apple's headset is more vibrate and wider than the competition. I would love to see that. would even mind the budget because for the unlimited amount of games and gaming potential here to explore. Face ID tech coming to hand gesturing is amazing.
  • Reply 7 of 10
    How will FaceTime (and similar) be able to show your face? Any ideas for solutions?
  • Reply 8 of 10
    Add those of us with compromised vision and it might take even longer before we will be able to retire our handsets.  
    Totally agree. I have custom lenses for my PSVR, Oculus Quest and Samsung Odyssey headsets. They make a massive difference in the quality and comfort of the VR experience. Every headset maker should be thinking about how to easily install custom lenses on their future headsets. 60% of the adult population need corrective lenses. Contact lenses are not an acceptable substitute for having corrective optics built into the headset.
  • Reply 9 of 10
    I love VR/AR headsets but they won't be replacing hand held devices for a long, long time if ever. If they got small and light enough, they could replace your computer screen and iPad but the iPhone is in a different category. I keep mine in my back pocket when I am doing yard work which includes swinging a pickaxe. No headset will replace it as I require some degree of eye protection and wont risk damaging either my eyes or an expensive gadget. This is just one personal example but there are many others. Can you use an AR headset while driving? I am guessing not or at least not until your car drives itself. Let's take a look at the technological roadblocks preventing constant use of an AR headset:
    Currently VR headsets are so big and heavy that they become uncomfortable to wear after an hour or two.
    Even if an AR headset were smaller, it would still need to be powered by batteries. Where do they go? In your pocket with a cord dangling down your neck?
    You can shrink the vision system but what about the rest of the electronics? It all generates heat so how to you dissipate it if the electronics are built into a pair of glasses?
    AR is great but for the best virtual experiences, you need full VR. Will the AR glasses have a full VR mode or will we need another headset for that?
    Many people are very sensitive to differences between motion they see with their eyes and what they feel with their sense of balance. How do you do VR without making people sick?
    Finally the cameras. You can't have AR without some cameras to help with the tracking and scene analysis. People threw a hissy fit over the original Google Glass AR glasses because they had cameras. There was at least one fight over them. Is everyone ready to accept a camera on everyone's face constantly recording and sending the video back to the Apple's servers? What happens when Apple starts scanning those videos for anything illegal to protect kids or something?
    The Magic Leap device (similar to Microsoft HoloLens) projects into the glass rather than a conventional monitor that emits back light. You can wear the glasses all day and you won’t be bothered by the light. You’ll be bothered by the frame long before that.
    I have worn both for my job which is being in the VR/AR industry. An issue both suffer from is very low field of view for projection (it’s like a stamp in the center of your glasses). I can only assume Apple solved that problem. 

    AR glasses work differently from VR glasses, which to your point use conventional monitors stuck in front of your eyes. I can take 30 minutes but then I’m completely done with them. 

    This is my objection to AR glasses; I got eye surgery because I hated to wear a frame on my nose all day, and my eyes couldn’t handle contact lenses nor did I wanted to fiddle with these things in my eyes. I just don’t want to wear something on my head other than sunglasses, so why should I go back to the situation I invested to leave?

    My second objection is that it competes with the phone which already does so much and has an amazing display. 
    It’ll have to give me so much more new and practical use-cases before I decide adapt to using AR glasses. And how will I move between AR glasses and my phone? Does that makes sense?

    My place for AR will be most likely a work environment (e.g work collaboration, learning) rather than gaming or consumer like setting that Apple has focused on (traditionally Apple does a very poor job of enterprise - Google’s services for work as an example is miles ahead of Apple’s).

    If the practical use-cases aren’t there it just becomes an Apple Watch for me; great to use to see the time and track my workouts, but not fulfilling its promise of leaving my phone at home more and being more ‘in the moment’ and leveraging third-party phone apps. 

    In a way I’m afraid it ends up like the Pro Max chip; an amazing SoC for editing video, music and photos, but now what I want to fully leverage the power and play some triple-A games with it I’m instead stuck with poor App Store offerings of Rosetta 2 / OpenGL translated of games that will never be updated to perform natively.
    Hardware without great software in a wide range of categories diminished the net usefulness of it.
  • Reply 10 of 10
    Add those of us with compromised vision and it might take even longer before we will be able to retire our handsets.  
    Totally agree. I have custom lenses for my PSVR, Oculus Quest and Samsung Odyssey headsets. They make a massive difference in the quality and comfort of the VR experience. Every headset maker should be thinking about how to easily install custom lenses on their future headsets. 60% of the adult population need corrective lenses. Contact lenses are not an acceptable substitute for having corrective optics built into the headset.
    There have been rumors of prescriptive lenses. https://www.tomsguide.com/amp/news/apple-glasses
Sign In or Register to comment.