Apple patent points to in-air 3D gesture UI for Apple TV, Mac and beyond
One month after receiving a patent reassignment from Israel-based 3D mapping R&D firm PrimeSense, Apple on Tuesday was granted another property detailing possible software implementations to go along with previously outlined motion-sensing hardware.
Source: USPTO
The U.S. Patent and Trademark Office assigned Apple U.S. Patent No. 8,933,876 for "Three dimensional user interface session control," or more plainly, a software UI for use with PrimeSense's motion-sensing hardware.
In practice, the patented system incorporates a motion and depth-sensing optical sensor, outlined in a previously reassigned patent, with proprietary software to create a three-dimensional non-tactile user interface capable of recognizing and translating user gestures into computer commands.
Specifically, the invention takes tracking, motion and depth input data from the camera/sensor setup and applies it to a 3D user interface. Users move their hand along multiple points, recognized in a 3D coordinate plane along X, Y and Z axes, to create gestures that are then processed by a host computer as app or system commands. In one example, a user can unlock the system UI with a rising gesture made along a vertical axis.
According to the patent, the device itself moves in between three distinct operating states, including any combination of locked/unlocked, tracked/not tracked and active/inactive. For example, if a sensing device is locked, it may also be ignoring user input -- not tracking -- to avoid inadvertent unlocking commands. In this scenario, the UI would also remain in an inactive state.
Users may alert the device to an incoming unlock command by performing a "focus gesture." These specialized movements engage the device, bringing it out of a not tracked/inactive state to locked/tracking/active. For example, a focus gesture could entail a "push" or "wave" of the hand that, when correctly recognized the sensor, enables motion tracking and activates the UI.
Alongside provisions for focus gestures is a method of "dropping sessions," which describes a graphical and operational solution to canceling user input. For example, in a scenario invoking a soft bar (shown in the illustration above as a system dock), a user raises their hand to unlock and activate motion tracking. If the user drops their hand out of the sensing hardware's field of view, the session is likewise dropped, with the example soft bar falling offscreen to reflect an inactive UI state.
Building on this point, Apple's patent notes that a user's gestures must be within the sensor's field of view, a seemingly obvious requirement, but one that may detract from a positive user experience if implemented incorrectly. The patent suggests an onboard light array be configured to illuminate when a user is positioned within its field of view, giving a visual cue that the system is ready for input.
Today's patent disclosure follows a December reassignment of another PrimeSense patent covering a 3D mapping and motion tracking device. The firm applied very similar infrared motion tracking technology when it helped Microsoft build the original Xbox Kinect sensor.
Apple purchased PrimeSense -- and its patents -- in 2013 as part of a deal worth somewhere between $345 million and $360 million. Following the acquisition, industry scuttlebutt purported Apple would likely incorporate motion-sensing hardware and software in a revised Apple TV unit, though the device has yet to be introduced.
Recent reports claim the release of Apple's next-gen set-top streamer is in limbo due to stalled talks with cable providers keen on keeping their content locked up behind onerous licensing terms.
Apple's 3D user interface patent was first filed for in December 2011 and credits Micha Galor, Jonathan Pokrass and Amir Hoffnung as its inventors.
Source: USPTO
The U.S. Patent and Trademark Office assigned Apple U.S. Patent No. 8,933,876 for "Three dimensional user interface session control," or more plainly, a software UI for use with PrimeSense's motion-sensing hardware.
In practice, the patented system incorporates a motion and depth-sensing optical sensor, outlined in a previously reassigned patent, with proprietary software to create a three-dimensional non-tactile user interface capable of recognizing and translating user gestures into computer commands.
Specifically, the invention takes tracking, motion and depth input data from the camera/sensor setup and applies it to a 3D user interface. Users move their hand along multiple points, recognized in a 3D coordinate plane along X, Y and Z axes, to create gestures that are then processed by a host computer as app or system commands. In one example, a user can unlock the system UI with a rising gesture made along a vertical axis.
According to the patent, the device itself moves in between three distinct operating states, including any combination of locked/unlocked, tracked/not tracked and active/inactive. For example, if a sensing device is locked, it may also be ignoring user input -- not tracking -- to avoid inadvertent unlocking commands. In this scenario, the UI would also remain in an inactive state.
Users may alert the device to an incoming unlock command by performing a "focus gesture." These specialized movements engage the device, bringing it out of a not tracked/inactive state to locked/tracking/active. For example, a focus gesture could entail a "push" or "wave" of the hand that, when correctly recognized the sensor, enables motion tracking and activates the UI.
Alongside provisions for focus gestures is a method of "dropping sessions," which describes a graphical and operational solution to canceling user input. For example, in a scenario invoking a soft bar (shown in the illustration above as a system dock), a user raises their hand to unlock and activate motion tracking. If the user drops their hand out of the sensing hardware's field of view, the session is likewise dropped, with the example soft bar falling offscreen to reflect an inactive UI state.
Building on this point, Apple's patent notes that a user's gestures must be within the sensor's field of view, a seemingly obvious requirement, but one that may detract from a positive user experience if implemented incorrectly. The patent suggests an onboard light array be configured to illuminate when a user is positioned within its field of view, giving a visual cue that the system is ready for input.
Today's patent disclosure follows a December reassignment of another PrimeSense patent covering a 3D mapping and motion tracking device. The firm applied very similar infrared motion tracking technology when it helped Microsoft build the original Xbox Kinect sensor.
Apple purchased PrimeSense -- and its patents -- in 2013 as part of a deal worth somewhere between $345 million and $360 million. Following the acquisition, industry scuttlebutt purported Apple would likely incorporate motion-sensing hardware and software in a revised Apple TV unit, though the device has yet to be introduced.
Recent reports claim the release of Apple's next-gen set-top streamer is in limbo due to stalled talks with cable providers keen on keeping their content locked up behind onerous licensing terms.
Apple's 3D user interface patent was first filed for in December 2011 and credits Micha Galor, Jonathan Pokrass and Amir Hoffnung as its inventors.
Comments
I just bought an LG Smart TV and the WebOS magic remote is a pretty good implementation.
I've never been a fan of 3D-gestures on smartphones. It was always too clunky and pointless, but on a TV I can see myself using it a lot.
Sounds like Xbox Kinect.
Not surprising. PrimeSense amde the sensors for the original Xbox 360 Kinect device
Connect licenses Apples technology.
EDIT: i'm wrong. But they used to in previous xBox releases.
I was wondering what Apple was up to with the PrimeSense IP.
I would love to see that UI implemented in a 10K TV screen by the end of the Summer.
The box on top of the TV sure isn't going to work.
The box on top of the TV sure isn't going to work.
Gotta love these generic images used for diagrams in patent announcements. What's up with that bulky PC tower / circa. 2012 Mac Pro connected to the TV / monitor, and the clunky looking external webcam?
LOL forget about all of that. Look at that crazy game of Pong!!! LOL
Apple technology? You mean PrimeSense which Apple bought?
I've never understood why patents granted are so newsworthy. Apple has over 20 patents on "smart pens" going back numerous years and yet we have nothing to show for it. Same thing with Apple's Smart Cover. Several patents on different Smart Cover innovations going back to 2011 but nothing realized in a product. I'd love to know how many things Apple patents actually make it into products for sale.
Apple technology? You mean PrimeSense which Apple bought?
I've never understood why patents granted are so newsworthy. Apple has over 20 patents on "smart pens" going back numerous years and yet we have nothing to show for it. Same thing with Apple's Smart Cover. Several patents on different Smart Cover innovations going back to 2011 but nothing realized in a product. I'd love to know how many things Apple patents actually make it into products for sale.
I meant both but I shortened it to "Apple Technology". There are plenty of patents owned by Apple not originated by Apple, but which are now being worked on by Apple employees. Multi-touch is an example.
In this case Apple will probably do something precisely because they spent a far whack of money on an acquisition. When is the question.
I've always been curious as to who does the drawings for patents...
I love them so much. I think they purposely draw it in a way that makes it look as dumb and pointless and clunky as possible, while still being technically accurate. Clearly this is for the AppleTV, but they show the "computer" as a 1990s tower and CRT monitor on a computer desk.
As for this, I love it as a TV interface. At my house, the remote is always getting misplaced in other rooms, between the couch cushions... rendering the AppleTV useless until it's found. I know you can connect the remote app, but it needs to be signed in to the same iCloud account and we each have different iCloud accounts. It would be nice to have an always-there interface.
The women are on the other end of the web chat they have going:
They just can't show that because of political correctness these days.
Sorry, my Swedish spellcheck completely botched my post.
I meant to write. " ... but where are the women?" You seem to have gotten the jist of what I was trying to say anyway.
A bit sexist to only have men in the illustrations
These are not the patents you're looking for.