How Apple's iPhone X TrueDepth AR waltzed ahead of Google's Tango

Posted:
in iPhone edited October 2017
Apple's iOS 11 brings Augmented Reality to a huge installed base of over 350 million iOS devices with powerful A9, A10 Fusion or A11 Bionic chips. However, another form of AR is about to be introduced, exclusive to the company's new iPhone X. Here's a deeper look at Apple's big AR surprise.

ARKit on your face with the TrueDepth camera

While Apple's existing platform for ARKit in iOS 11 targets a huge installed base of single-camera iPhone and iPad models, the company is also about to release something that no Android phone will have in the short term: a front facing depth camera capable of performing even more sophisticated AR effects.

On iPhone X, ARKit apps can take advantage of its TrueDepth camera to make the user's face into the surface where 3D graphics are built. This uses facial geometry to track the user's pose, face shape and their expression, enabling Animoji (below), Portrait Lighting effects, masks and avatars; effectively, a very sophisticated new depth of AR effects for the front-facing camera.



Rather than just putting an enhancement layer on top of a photograph, Face Tracking in ARKit detects the user's face and movements and tracks positioning using its array of cameras and the phone's 6-axis motion sensors. This can be used to create "selfie effects" such as applying virtual makeup or tattoos, facial hair, glasses, jewelry or hats. It can also be used to perform "face capture," which takes real-time facial expressions to animate a 3D model such as a character avatar.

Google's AR was Ahead before it fell so faR behind

Apple's new level of depth camera-based AR technology will begin building out a platform of tens of millions of iPhone X users just as Android's ARCore trickles out basic single-camera AR to a select few, premium-priced Android models.

Google isn't just a couple years behind in deploying AR software; the Android ecosystem is vastly behind in even creating a business model that could support the development of a mass market device with a depth camera capable of delivering the depth-based AR Apple is about to roll out to millions within the next month.

Oddly enough, Google was originally ahead of the game with its experimental Project Tango, a more ambitious form of AR that required a depth camera. The technology traces its lineage back to the same work at PrimeSense and FlyBy Media, both of which Apple acquired.

In addition to performing Visual Inertial Odometry, Tango also addresses SLAM (Simultaneous Localization And Mapping), which builds out a map of what the depth camera system "perceives" for later use. This growing map of your home and everywhere else you use Tango collects an expansive model of the world, something like Google's camera cars that create street views for Maps. Not including SLAM in ARKit was an intentional decision

ARKit 1.0 doesn't attempt to map out the world as you use it, focusing instead on lightweight, immediate uses of AR to create explorable 3D visuals. However, it can be used in conjunction with GPS or iBeacon indoor location systems to provide AR in combination with known maps. Not including SLAM in ARKit was an intentional decision, because Apple has acquired companies working with interior mapping, specifically using SLAM techniques.

The greater technical complexity Google worked on in Tango, combined with its prerequisite, complex and expensive depth cameras to work at all, added up to a project that didn't ship in a workable form ordinary people could use. Further, rather than building out the hardware and software technology to deploy AR to the masses itself, Google relied on third parties to find some application of the experimental work it was doing.

Google's Tango danced in the wrong direction!

Across years of work, Google dribbled out an experimental Tango phone and tablet to about 3,000 curious researchers before teaming up with China's Lenovo to install a depth camera system on the back of the Phab 2 Pro "Tango" phone last year, followed by a similar Asus-built Zenfone AR earlier this year.


Lenovo Phab 2 Pro Tango phone


However, both of those Tango phones implemented depth-AR using the rear camera. That's a design decision that appears to makes sense, but it renders depth-AR as a standalone gimmick that's only marginally better than single camera AR using VIO, despite making the phone significantly more expensive. CNET concluded that the Phab 2 Pro was "a bulky, heavy, mediocre handset with disappointing hardware, an old version of Android and no NFC."

Unlike Tango phones, iPhone X implements a front-facing TrueDepth sensor. Compared to the status quo, that seems backwards until you realize that, in addition to doing depth-based AR as a neat "trick," it also positions the depth sensor to do other valuable things: Face ID authentication, familiar Portrait Lighting effects on selfies, and an advanced type of face-based avatar filters that are already extremely popular among young Snapchat users.

Another reason why Apple's front-facing TrueDepth camera has significant mobile advantages over the rear-mounted depth camera system of Tango: depth cameras use a lot energy, so expecting to use one to navigate through a store as your phone is held out in front of you (Tango's "Visual Positioning System" was actually one of the features Google presented at IO this summer!) isn't really sensible. Further, using such a depth camera outside can be problematic because sunlight can overwhelm the IR illumination and dots that are projected by depth camera systems.

By putting its TrueDepth sensor on the front, Apple can turn it on only when needed (with frequent short bursts for Face ID, or only when users are actively recording Animoji). The system also only has to illuminate your face, rather than attempting to invisibly light up the entire environment.

Face ID


Note that Apple points out that Face ID works "even in the dark," highlighting its engineering intent for use where a user would expect to have it work, in contrast to Tango and Google's VPS, which ambitiously pretend to work everywhere imaginable, in large, open and outdoor environments that are simply much harder to actually deliver useful AR-related SLAM or VIO as practical, functional features.

Our new #Tango-enabled Visual Positioning Service helps mobile devices quickly and accurately understand their location indoors. #io17 pic.twitter.com/1pYlCGM8eg

-- Google (@Google)

Tango is a Newton. ARKit is an iPod

A big part of successfully deploying any new technology (or art) is finding a rooting in the familiar. If it's too new, nobody knows how to relate to it. However, if it's too common, there's little proprietary value in developing the technology. Apple commonly aims, not just to be first, but to hit the market with a practical, valuable solution in a way that nobody else has thought to deliver.

Apple hasn't always been so focused. In the early 1990s, it developed Newton as an ambitious tablet project, but didn't complete enough of the work to make it clearly and obviously useful at the price its new technology demanded. After years of work, Newton was essentially blindsided by Palm, which shipped a cheaper, less technically impressive but more practical, useful Pilot PDA.

Google has a similar problem with Tango: it's put lots of effort into developing technology that Apple intentionally chose not to implement (such as the SLAM mapping that plays a big role in Tango but isn't in ARKit 1.0), resulting in complexity and expense without a parallel increase in value for most mainstream users.

Academics can blog about how much more superior Tango is on paper, but if it can't deliver on its ambitious promises, and if nobody is using it because it's expensive to implement, and it never really gets updated and maintained and finished because of that lack of use in the real world, it doesn't really matter.

Hidden as plain as the notch on your face

In the same way that the bloggers who stole a prototype iPhone 4 back in 2010 had it in their hands but didn't even know it would feature a Retina Display or a gyroscope or introduce FaceTime, the basic design of iPhone X was splashed around by leakers who didn't really grasp that Apple's depth AR strategy was more than just a way to visualize Ikea furniture in a room.



TrueDepth was implemented as a way to securely identify and recognize the device's owner, enhance their selfies and pull them into an animated world of Anaimoji chats and third-party avatars. Apple's novel implementation of TrueDepth on iPhone X makes the user the focus of its unique new depth camera

There are some interesting applications for depth-based AR (requiring an actual depth camera, unlike most of ARKit) on the back of a mobile device. But there are already commercial solutions offering this (including Occipital's $379 external Structure Sensor for iPad) and they haven't exactly taken the world by storm.

Apple's novel implementation of TrueDepth on iPhone X makes the user the focus of its unique new depth camera. It combines a flood illuminator that reflects invisible infrared light back to the infrared camera, in parallel with a dot projector that creates a grid of landmark points to serve as a structure sensor.

Multiple paths to implement depth-based AR

There are a lot of different ways to implement this technology: Microsoft focused on IR mapping of the living room in front of Xbox Kinect games to track users' bodies. While novel, it hasn't delivered gameplay that's superior to other forms of motion controllers, and has lots of simplistic drawbacks. The system also adds expense, playing a role in making Microsoft's latest console generation too expensive to effectively compete until the company stripped out Kinect as a bundled feature.

Similarly, Google's Tango experiments that put a structure sensor on the back of big Android phones offered some potential for novel features related to mapping out objects in the room, but was too expensive to find an audience for what little it could actually do out of the box.

Apple's two-pronged approach brings single camera ARKit to an installed base of 380 million devices, acquainting users with the technology in a free update to iOS 11. In parallel, users can opt into the even more sophisticated world of front facing, TrueDepth camera AR on iPhone X, which looks at the user rather than the outside world behind the device.

And while some sought to ridicule Apple over the frivolity of demonstrating how the TrueDepth camera could be used to animate emoji, in reality Craig Federighi spent less time showing the feature off for iPhone X than Steve Jobs spent demonstrating how iPhone 4's gyroscope could be used to animate Jinga blocks. To intelligent observers, both were examples of how incredible new technology could be put to use by app developers once delivered to a large platform of users.

Outsmarting Google with secret development

Apple's efforts to find useful, attractive features for its TrueDepth system and use them to sell iPhone X in large volumes is so obvious in hindsight that Android makers are now rushing to copy it. This explains why Apple didn't brag about its implementation plans for TrueDepth AR until it had iPhone X nearly ready to sell.

Further, Apple's strategy for rolling out single camera AR to over 380 million users in iOS 11 while introducing its depth-based Face Tracking AR as an exclusive feature of iPhone X is only obvious now, far too late for copycat Android makers or their platform leader to quickly catch up.

Google has attempted to salvage parts of its depth-camera Tango work under a simpler single-camera implementation it recently rebranded as ARCore. But by failing to identify the most valuable and easiest to deploy technologies first, in tandem with developing a more advanced layer that could be sold as an exclusive premium, it's now far behind Apple's ARKit rollout and must start over from scratch on its depth-based AR, if it can even find anyone to adopt its second attempt at premium-priced Tango, modeled after Apple's work.

Apple's secrecy in developing a unique, innovative way to apply its TrueDepth technology essentially replicated the history of Google Wallet (steamrolled by Apple Pay), Android NFC Beam (run over by Bluetooth 4 Continuity) and Android fingerprint sensors (trounced by Touch ID). Being first with experimental new tech was clearly not as a valuable as going bigger with a better implementation that could offer real value to buyers.
radarthekatpatchythepiratepropodwatto_cobrawlym
«1

Comments

  • Reply 1 of 32
    calicali Posts: 3,495member
    Popcorn ready and one-post moron ready. 
    RacerhomieXcornchipSomeGuyInSFStrangeDayspscooter63watto_cobralolliverRayz2016jony0
  • Reply 2 of 32
    Can’t wait for the firestorm of Google defense. 
    cornchipSomeGuyInSFStrangeDayswatto_cobralolliver
  • Reply 3 of 32
    Fully strapped in and waiting for the fun to begin !
    cornchipwatto_cobralolliver
  • Reply 4 of 32
    RacerhomieXRacerhomieX Posts: 95unconfirmed, member
    Every word is true here, Google has failed in AR. Just like in Wearables. No one even talks of android wear now.
    LukeCagewatto_cobralolliverjony0
  • Reply 5 of 32
    Waiting for the AdDroid settlers to start howling.....3, 2, 1.......lol
    patchythepiratecornchipStrangeDayswatto_cobra
  • Reply 6 of 32
    Great article, but it was “Jenga” not “Jinga” that Steve Jobs used in his demo. 
    cornchipwatto_cobra
  • Reply 7 of 32
    You have to give Apple credit for seeding the market, but the technology is absolutely useless at moment.

    Smart people will be buying the IPhone 8 over the IPhone X.  Pocketing the price difference, and prepare for the 2.0 version available in a few years (when apps become available) that actually does something usefull.

    Apple’s strategy will be very difficult for Google to duplicate.  Google could do it in the Google branded phones, but will struggle getting their partners on board.  I doubt many will accept financial loses to push the technology out to early adopters.

    AR is a ‘chicken or egg’ situation only with ‘hardware or software’.

    The next thing Apple needs to do is offer a promotion to app developers (to forgo their ‘cut’ from the Apple App Store for 2 years) for apps that utilize AR.
    cornchip
  • Reply 8 of 32
    radarthekatradarthekat Posts: 2,428moderator
    You have to give Apple credit for seeding the market, but the technology is absolutely useless at moment.

    Smart people will be buying the IPhone 8 over the IPhone X.  Pocketing the price difference, and prepare for the 2.0 version available in a few years (when apps become available) that actually does something usefull.

    Apple’s strategy will be very difficult for Google to duplicate.  Google could do it in the Google branded phones, but will struggle getting their partners on board.  I doubt many will accept financial loses to push the technology out to early adopters.

    AR is a ‘chicken or egg’ situation only with ‘hardware or software’.

    The next thing Apple needs to do is offer a promotion to app developers (to forgo their ‘cut’ from the Apple App Store for 2 years) for apps that utilize AR.
    Many app developers will view AR as a means to differentiate.  The large number of AR-compatible phones already in existence provides a readymade market.  The X has other features that will tempt many buyers, even if they wouldn’t purchase solely for the front-facing AR capabilities.  They’ll get those essentially for free while purchasing the phone for its edge-to-edge display and other features.  
    bobby88watto_cobralolliver
  • Reply 9 of 32
    You have to give Apple credit for seeding the market, but the technology is absolutely useless at moment.

    Smart people will be buying the IPhone 8 over the IPhone X.  Pocketing the price difference, and prepare for the 2.0 version available in a few years (when apps become available) that actually does something usefull.

    Apple’s strategy will be very difficult for Google to duplicate.  Google could do it in the Google branded phones, but will struggle getting their partners on board.  I doubt many will accept financial loses to push the technology out to early adopters.

    AR is a ‘chicken or egg’ situation only with ‘hardware or software’.

    The next thing Apple needs to do is offer a promotion to app developers (to forgo their ‘cut’ from the Apple App Store for 2 years) for apps that utilize AR.
    You’re under the assumption that Apple needs to get developers on board.  In the modern app landscape, you roll out new features and innovate, or fail.  Apple being the largest AR market, developers will rush to add AR support to apps that don’t need it, in an attempt to secure downloads.  I personally think the uneccessary deployment of AR will be chaotic.  AR is not the best way to do everything, but it’s deffinately a selling point to gain market traction.

    It’s exciting to imagine what creative new services and utilities will be engendered by such a leap.  Kids of today will grow up in a world as dominated by AR/VR as ours was by the internet.  I wonder how thin of a line will separate the real and virtual worlds 10 years form now.
    radarthekatcornchiplolliverstantheman
  • Reply 10 of 32
    When I said ‘promotion for AR’ I didn’t mean slapping ‘lipstick on a pig’ to take advantage of a promotion.  The use of AR in an App has to be significant...

    I don’t know how AR will be used, but I expect that significant costs will be involved.  Apple should do everything possible to get those Apps written for Apple’s ecosystem to differentiate from Android.

    I don’t see a ‘killer’ App for AR (from the article) and nothing that exists today would justify the price premium of X.  If anything these phones are vastly overpowered for what they’re use for.

    $1000 for device that makes phone calls, etc. is ridiculous.  AR has the potential to use some of the capability of the phone, if it goes unused it’s a waste.  Therefore expediting AR with a promotion makes sense.  It would cost Apple practically nothing and give them a multi year lead.


  • Reply 11 of 32
    MacProMacPro Posts: 17,382member
     No in depth rebuttal in defense of Google so far.  Gatorguy must have the night off.
    suddenly newtoncornchipStrangeDayswatto_cobralolliverRayz2016
  • Reply 12 of 32
    NY1822NY1822 Posts: 495member

    it literally has got to the point of Google copying Apple right down to an aerial shot of a pink glazed donut with rainbow sprinkles! 
    (not to mention their milenial sounding female announcer making stupid jokes, a blatant ripoff from Apple)
    edited October 2017 cornchipwatto_cobra
  • Reply 13 of 32
    foggyhillfoggyhill Posts: 4,692member
    When I said ‘promotion for AR’ I didn’t mean slapping ‘lipstick on a pig’ to take advantage of a promotion.  The use of AR in an App has to be significant...

    I don’t know how AR will be used, but I expect that significant costs will be involved.  Apple should do everything possible to get those Apps written for Apple’s ecosystem to differentiate from Android.

    I don’t see a ‘killer’ App for AR (from the article) and nothing that exists today would justify the price premium of X.  If anything these phones are vastly overpowered for what they’re use for.

    $1000 for device that makes phone calls, etc. is ridiculous.  AR has the potential to use some of the capability of the phone, if it goes unused it’s a waste.  Therefore expediting AR with a promotion makes sense.  It would cost Apple practically nothing and give them a multi year lead.


    Apple providing the API to make it easy to write AR and the hardware and OS that supports those functions in great number both "paying people to write the apps" and making sure there are "people to buy them".  Then, it all comes down to capitalism and creativity. There will be 1000 AR apps, 900 of them in which AR will be either an useless adjunct or the app itself will be useless. But, 100 of them will be an interesting use and 20 of them will be simply brilliant.

    Of those brilliant use, 2-3 will emerge as being able to gain major mainstream traction from those apps and build themselves a brand name.

    A good chunk of the interesting and brilliant apps will either be by existing players, or be bought by major current players like Apple, Google, Amazon, Microsoft or Facebook.

    That's how it went for the apps market, or almost any market really.

    Apple has already made its part to make it happen.

    cornchipwatto_cobralolliver
  • Reply 14 of 32


    Face ID




    A bit off topic, but most every time I see this picture, I find myself pausing to admire it. The overall composition, with Phil Schiller against the floating backdrop image of a woman lit by the representational glow of a camera sensing her face is very captivating.
    teejay2012cornchipwelshdogd_2watto_cobralolliverRayz2016
  • Reply 15 of 32
    From the AR apps I've seen so far, there doesn't seem to be anything that will help Apple sell more iPhones. Of course, that's probably not Apple's intentions. I'm only taking a personal near-term look at the technology and there may be a lot more to AR than I realize. I just don't see hordes of people rushing out to buy iPhones to play with the simple AR apps that are now available.
  • Reply 16 of 32
    With Google the tech is all about finding where you are and then slinging adverts to you.

    So you are standing outside a KFC with your 6yr old and suddenly a targetted ad appears offering you 10% off a big Mac and a Happy meal at the 'place' just down the road.
    It knows that you have your 6yr old with you and exactly where you are because of the rear facing camera and that everything is tied into location services. Your Android phone is their spy in your hands.

    Contrast that to Apple's approach.
    Only used when needed. etc etc
    It is all there in the article.
    Different requirements means different solutions.

    Personally I don't have much use for AR and absoulutely none for VR so at this stage of development, I'm certainly in the MEH camp. Maybe that will change. I don't know about the future though.
    I will wait for the 2019 phones before even considering upgrading but the article was well worth reading. Thanks.

    lolliver
  • Reply 17 of 32
    When I said ‘promotion for AR’ I didn’t mean slapping ‘lipstick on a pig’ to take advantage of a promotion.  The use of AR in an App has to be significant...

    I don’t know how AR will be used, but I expect that significant costs will be involved.  Apple should do everything possible to get those Apps written for Apple’s ecosystem to differentiate from Android.

    I don’t see a ‘killer’ App for AR (from the article) and nothing that exists today would justify the price premium of X.  If anything these phones are vastly overpowered for what they’re use for.

    $1000 for device that makes phone calls, etc. is ridiculous.  AR has the potential to use some of the capability of the phone, if it goes unused it’s a waste.  Therefore expediting AR with a promotion makes sense.  It would cost Apple practically nothing and give them a multi year lead.


    I have no numbers to back this up, but I suspect that a very small percentage of iPhone activity is used to make phone calls.  

    I paid about $2,500 for an Apple ][  (and accessories) in 1978 -- that's approximately $10,000 on 2017 dollars.

    The $1,000 iPhone X is many times more capable than that $10,000 Apple ][ -- and when, however infrequently, the need arises, it can even make phone calls.
    d_2StrangeDayspscooter63watto_cobralolliverRayz2016
  • Reply 18 of 32
    Fully strapped in and waiting for the fun to begin !
    So you’re admitting these articles are just click bait. ;)
    gatorguymuthuk_vanalingam
  • Reply 19 of 32
    koopkoop Posts: 337member
    MacPro said:
     No in depth rebuttal in defense of Google so far.  Gatorguy must have the night off.
    To be honest AR is dull. I've seen the early stuff, and some of the popular games. I don't know why people and DED lose their minds over Google like this. We live in a duopoly, both companies are just gonna chug out massive earnings reports and Google's gonna put all its stuff on iPhone and the consumer gets to enjoy the rising prices of phones, encroachment of privacy and all these undercooked tech showcases. These guys barely compete with each other. They just need to match features and call it a day.
    gatorguymuthuk_vanalingam
  • Reply 20 of 32
    Fully strapped in and waiting for the fun to begin !
    So you’re admitting these articles are just click bait. ;)
    Lol! Not at all. They are argumentative essays, which inevitably attract the other side of the argument. 
    watto_cobralolliver
Sign In or Register to comment.