Apple patent details visual-based AR navigation, confirms Flyby Media acquisition

Posted:
in General Discussion edited August 2016
Apple on Tuesday was granted a patent detailing a method of device localization -- mapping -- using computer vision and inertial measurement sensors, one of the first inventions to be reassigned from this year's acquisition of augmented reality startup Flyby Media.




The patent, published by the U.S. Patent and Trademark Office as "Visual-based inertial navigation," allows for a consumer device to place itself in three-dimensional space using camera imagery and sensor data.

Specifically, the system marry images from an onboard camera or cameras with corresponding data from gyroscope and accelerometer hardware, among other sensors, to obtain information about a device's position at a given point in time. Subsequent images and measurements are compared and contrasted to determine device position and orientation in real time.

As noted in the patent, visual-based inertial navigation systems do not require GPS signals or cell towers to achieve accuracy down to mere centimeters. However, tracking past variables often calls for images and sensor measurements to be stored for later processing, a method requiring computational overhead unfit for mobile hardware.

To overcome such obstacles, Apple's invention implements a sliding window inverse filter (SWF) module designed to minimize computational load. The technology first processes information from overlapping windows of images as they are captured by a device camera, then tracks features in the images with corresponding sensor data to determine an estimated device state.




Next, the SWF estimates the position and orientation of objects proximate to the device. Estimates for device state and proximate objects are then calculated for each image in a series, or window, of images. Finally, the SWF summarizes each window of images, converting the estimates into information about the device at a single point in time. Summarized information about one window can by applied to processing steps in a second, overlapping, window.

In a real world situation, SWF modules might be used to power augmented reality navigation solutions. For example, SWF position and orientation estimates might be used to label points on a digital map or act as a visual aid in locating items in a retail store. Since the module is capable of maintaining the last known position of an object, it can lead users back to a misplaced set of keys or other important items.

One potential implementation incorporates depth sensors to create a 3D map of a given environment, while another integrates a device's wireless radios to mark signal strength as a user walks through a building.




It is unclear if Apple has plans to integrate SWF technology into an upcoming product, but the company is said to be hard at work on augmented and virtual reality solutions.

Apple's visual-based inertial navigation patent was first filed in 2013 and credits former Flyby Media employees Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko and Oriel Bergig, as well as University of Minnesota professor Stergios Roumeliotis, as its inventors.

Earlier this year reports claimed Apple purchased Flyby Media, a startup focused on augmented reality solutions linked to Google's Project Tango. Of the inventors named in today's patent, Naroditsky and Grygorenko now work at Apple.

Comments

  • Reply 1 of 8
    The next few years will be exciting, to see where Apple implements AR. Possibilities:
      1. Gaming ------------------- true console with AR capabilities
      2. Health --------------------- auto-label medical videos, photos
      3. Emergency Response --- help rescuers quickly assess a situation
      4. Car ------------------------ AR-enabled windows to label what you're passing

    Just idle speculation, of course, but Apple has many outlets into which they could apply AR.
    stanthemanmejsricSoli
  • Reply 3 of 8
    jakebjakeb Posts: 557member
    The real killer app for this might not even be an app that Apple makes, but being able to provide a relatively simple API for this kind of navigation to developers. Not needing to sort out the mapping of the physical surroundings yourself opens up AR to a much bigger pool of developers and would give iOS a big advantage over Android. 
  • Reply 4 of 8
    jakeb said:
    The real killer app for this might not even be an app that Apple makes, but being able to provide a relatively simple API for this kind of navigation to developers. Not needing to sort out the mapping of the physical surroundings yourself opens up AR to a much bigger pool of developers and would give iOS a big advantage over Android. 
    ....yes, and along with a variety of other real-world uses.

    This seems like a successfully strategy for Apple in marketing new products which is to:
     - build HARDWARE in coordination with the software PLATFORM
     - RESULTING in a tool that performs useful/fun/creative tasks effectively
     - MARKET accordingly, for example... instead of touting specs or some novel feature, Apple emphasizes at least a few developed applications  that fully utilizes hardware/software that other appliances cannot do (as well).  We'll see this strategy at iPhone7 event.

    Some criticize Apple for lack of innovation.  And while this is arguable, it is not arguable that Apple is doing a lot of behind-the-scenes innovation that requires the coordination of the entire platform, not just one device, one hardware spec or one API. 


  • Reply 5 of 8
    Find my iPhone would be sweet with my iPad showing a 3D map of my house, leading me to its precise location!
    patchythepirate
  • Reply 6 of 8
    SpamSandwichSpamSandwich Posts: 29,748member
    Reminder:  A patent ≠ A product
  • Reply 7 of 8

    As others have posted, this could be a big deal.

    Such things as the  ability to:
    • create a 3D map of your home for private use (decorating, security, HomeKit, etc.)
    • refine your last GPS location down to centimeters -- you are here
    • relate your current exact location with other, known/published general locations -- mall, airport, supermarket, golf course, 911, etc. -- how do I get there, what club should I use, or where's the Hoisin sauce?
    • use an inexpensive device to locate a lost pet, child or aged relative

    But, as @SS says: Reminder:  A patent ≠ A product

    Edit:  As good as it sounds, there are some negative implications -- our always on camera would be capturing 3D images of people as well as places... will we all become Glassholes?

    edited August 2016
  • Reply 8 of 8
    "the system marry images…"

    You probably meant, "the system marries images." No? 
    :) 
    edited August 2016
Sign In or Register to comment.