Apple exploring Kinect-like 3D input for controlling Macs

Posted:
in Future Apple Hardware edited January 2014
Apple has shown interest in an entirely new way for users to interact with and control their Mac, with a new system allowing users to perform gestures with their hands in a three-dimensional space.



The concept was disclosed this week in a U.S. patent application discovered by AppleInsider entitled "Three-Dimensional Imaging and Display System." It describes a system akin to Microsoft's Xbox Kinect platform, which allows for controller-free interaction with devices.



Apple's solution would optically detect a user's hands and fingertips and measure their movements. In this manner, users could use their hands to control a Mac without the need for a mouse or keyboard.



In its application, Apple notes that current computer input devices like the mouse allow users to control a system in two dimensions, along the 'X' axis and 'Y' axis. But when manipulating objects in three dimensions, input methods like a joystick or mouse can be cumbersome.



"A need thus remains for uncomplicated, economical, yet highly effective 3D input devices for computers," the filing reads. "Such devices need to be able to detect, analyze and measure objects located in a 3D volume, and to observe and track any motions thereof."



Apple's filing notes that for any system to be successful, it needs to be "economical, small, portable and versatile." It must also provide a user with "integral, immediate feedback," and it needs to be adapted well for use with a number of devices, ranging from full-fledged Macs to portable electronics like the iPhone.



The hardware described by Apple in the application would be capable of tracking a user's head as well as their hands. Potential components include an infrared sensor or visible laser, high-speed photo detector, digital signal processor, dual-axis scanning device, and subsystems for analog and video.







Apple envisions a user interface that would allow a person to more naturally use their hands and fingertips to control a device. An accompanying display could show knobs, sliders and buttons that a user could virtually manipulate in a three-dimensional space.



In one image accompanying the application, a user is shown using both of their hands to select inputs on an iMac. With their left hand, the user is twisting a virtual knob, while on the right hand, their index finger is being used to press a button.



The system would aid the user with on-screen visual cues, allowing them to more easily manipulate objects on the screen. For example, Apple's system includes a virtual representation of the user's hand displayed on the screen of the iMac.



In the application, Apple provides a number of potential uses for its described system. For example, head tracking could be used to tilt an image or zoom in on the screen based on the location of a user's head. Accordingly, moving person could be followed and tracked with a motorized camera system.



With hand gesturing, the system could track individual elements including the fingers, thumb and palm. Using gestures, a user could accomplish complex tasks like 3D rotate, zoom, scroll and volume control.



Another feature of the system described by Apple is user presence detection. With this, the system could detect whether a user is sitting in front of a display, and could even identify unique users. The system could also automatically shut down and save power when the user walks away.







Other potential uses listed by Apple are surveillance, bar code reading, object measuring, image substitution or replacement, or a virtual keyboard. In the keyboard example, Apple notes that individual fingers could be tracked for typing, and the image of a keyboard could be projected onto a desktop to give users a visual aid while typing.



The proposed invention, made public this week by the U.S. Patent and Trademark Office, was first filed by Apple in August of this year. It is credited to Christoph H. Krah.
«1

Comments

  • Reply 1 of 40
    daharderdaharder Posts: 1,580member
    Nice and all, but how's this 'entirely new' if MS' Kinect already allows "users to perform gestures with their hands in a three-dimensional space" as a substitute for device input/controllers?
  • Reply 2 of 40
    Ever since Minoriry Report I have wanted to have something like this.
  • Reply 3 of 40
    Notice how old this is? That iMac is from the early 2000's.
  • Reply 4 of 40
    Quote:
    Originally Posted by [Greg] View Post


    Notice how old this is? That iMac is from the early 2000's.



    Is this the predecessor comment to saying Microsoft must have copied Apple again?
  • Reply 5 of 40
    nairbnairb Posts: 253member
    Quote:
    Originally Posted by AHrubik View Post


    Is this the predecessor comment to saying Microsoft must have copied Apple again?



    It is obvious that Apple thought of it first. Dont you know that apple have a patent on thinking of ideas before patenting them?
  • Reply 6 of 40
    Quote:
    Originally Posted by DaHarder View Post


    Nice and all, but how's this 'entirely new' if MS' Kinect already allows "users to perform gestures with their hands in a three-dimensional space" as a substitute for device input/controllers?



    It just is!!! Anyone that know anything about Apple knows that they invented this in August and were the first to invent it.



    All the similar things like:



    - People making this sort of thing a few years ago using Nintend Wii controllers

    - Microsofts well known plans for this going back a few years, releasing Kinect last year and the SDK for Windows at the start of this year

    - My Sony Vaio that I bought earlier this year accepting hand gestures using the web cam



    are all copy's of Apple's patent filed this year after all of them existing.



    If you can see how they copies Apple your just an idiot!!!
  • Reply 7 of 40
    Too bad Apple didn't purchase the technology in Kinect since it was offered to them first.
  • Reply 8 of 40
    blitz1blitz1 Posts: 433member
    Nothing new. It already exists and it is called Sixth Sense



    http://www.pranavmistry.com/projects/sixthsense



    Looks like another copy to me
  • Reply 9 of 40
    I'd turn it off instantly.

    Terrible, terrible idea. No matter who implements it. At least touch screens have some kind of tactile feedback.
  • Reply 10 of 40
    Quote:
    Originally Posted by OriginalG View Post


    Too bad Apple didn't purchase the technology in Kinect since it was offered to them first.



    Microsoft did not invent the motion controls in Kinect, they licensed the technology from Prime Sense. Prime Sense is not owned by or controlled by Microsoft and licenses its technology to any company willing to pay.
  • Reply 11 of 40
    Quote:
    Originally Posted by DaHarder View Post


    Nice and all, but how's this 'entirely new' if MS' Kinect already allows "users to perform gestures with their hands in a three-dimensional space" as a substitute for device input/controllers?



    Who said it was "entirely new"? Are you making up quotes or just building a straw man?
  • Reply 12 of 40
    charlitunacharlituna Posts: 7,217member
    Quote:
    Originally Posted by Doctor David View Post


    Who said it was "entirely new"? Are you making up quotes or just building a straw man?



    DaHarder is just being himself. You'll learn to ignore him like pretty much everyone else does
  • Reply 13 of 40
    Quote:
    Originally Posted by charlituna View Post


    DaHarder is just being himself. You'll learn to ignore him like pretty much everyone else does



    I should have learned long ago I'm sure. Back when he said he was buying 3 tablets for each member of his family.
  • Reply 14 of 40
    zoetmbzoetmb Posts: 2,654member
    Quote:
    Originally Posted by DaHarder View Post


    Nice and all, but how's this 'entirely new' if MS' Kinect already allows "users to perform gestures with their hands in a three-dimensional space" as a substitute for device input/controllers?



    Because ideas like this are not supposed to be patentable, only the specific implementation of the idea. And while I will admit that the patent office violates that concept all the time (like permitting Amazon to patent "one-click", which Apple then had to license from Amazon), it's how patents are supposed to work. Otherwise, "information is presented on a screen" would be patentable.
  • Reply 15 of 40
    This is exciting! While there are already numerous theories and products that already implemented in what I called 'air gestures'*, do be reminded that this is Apple we're talking about here - a corporation that innovates. Many of Apple products are already created physically and in theory, from the mouse to the mobile phone. However, none of these things are met with real-world success, most are just gimmicks or too bulky or without any design taste. Whenever Apple are interested or developing these things, it literally changed the world.



    Well, at least when the late Steve Jobs was around.



    Hopefully it won't need any extra cameras such as those employed by M$ and previously Sony with their PS2 EyeToy. I'd seen Kinect working but it isn't like this - this one will enable us to 'feel' the objects rather than just waving our hands to interact slides.



    As for Project Sixth Sense, it uses a wearable finger band and a camera - the Apple one looks like it only requires a board. Project Sixth Sense would rather make more sense if Sony would invest in them to advance their Move gadget (really, it's design is embarrassing - it currently looks like a microphone or worse, a dildo).



    Apple's track record in touches, both in performance and commercial results, is great. Surely, everyone remembers the multi-gesture trackpad on the Macbook Pro? I think that was before touch-screens was popular. In fact, touch screens became popular AFTER Apple innovate them - there were touch screens invented before but it never really took off! Apple had just put a gas propeller on that and then some!



    Now imagine if Apple pulled this air gesture gadget off:



    - 3D models can be sculpted, modelled and move around freely

    - Drawing on canvas will be easier - no more worrying about smudges or stylus pen nibs exhausted

    - Games will be more interactive; we saw these on the iPad and iPhone, now imagine doing so WITHOUT touching the screen or anything!

    - Web browsing will also be enhanced

    - HECK, a lot of possibilities I can't even think of!



    And because this is Apple, quality is assured. It does not only works but it works smoothly. As always, every new piece of tech has it's flaws but, again, because this is Apple, people are going to be more confident in buying and then using them.



    I know I am.



    * based on that we are just waving our hands and moving our body without touching anything
  • Reply 16 of 40
    asciiascii Posts: 5,936member
    Head tracking would be good for a computer. Not sure people would want to take their hands off the kb and mouse to do hand gestures. Maybe just left and right air-swipes to change app, similar to iPad. Or you could jerk your head left and right to switch app.



    Where gestures would really come in to their own is controlling the TV. But MS is way ahead on this one.
  • Reply 17 of 40
    Quote:
    Originally Posted by zoetmb View Post


    Because ideas like this are not supposed to be patentable, only the specific implementation of the idea. And while I will admit that the patent office violates that concept all the time (like permitting Amazon to patent "one-click", which Apple then had to license from Amazon), it's how patents are supposed to work. Otherwise, "information is presented on a screen" would be patentable.



    Patents, including the "one click purchase system" by Amazon which is actually called "METHOD AND SYSTEM FOR PLACING A PURCHASE ORDER VIA A COMMUNICATIONS NETWORK", require a very detailed explanation to accomplishing your task. Amazon had to explain a method and system to accomplish that task, so if another person comes out with a different process or system to accomplish the same task, Amazon can't sue. And there certainly are many other ways to accomplish a one click purchase, it's just that Apple decided to go with Amazon's solution.



    Amazon had to submit a patent that begins with explaining that there are many clients connected to the internet which are browsing Amazon's site, etc.. . So you can imagine the details required to explain the exact process in accomplishing the one click purchase.



    You can see the 18 page document here.
  • Reply 18 of 40
    Quote:
    Originally Posted by [Greg] View Post


    Notice how old this is? That iMac is from the early 2000's.



    I know, strange this would just be filed now with the diagram showing an early G4 iMac (notice the screen size/ratio). They obviously were working on this back in 2001/2002 (I think this iMac was introduced at Macworld SF 2002). But why submit the patent in August of 2011, nearly 10 years later? strange indeed.



    Quote:
    Originally Posted by AHrubik View Post


    Is this the predecessor comment to saying Microsoft must have copied Apple again?



    Well, no. MS has something on the market now, so Microsoft wins this round in the to production race. Having just gotten and Xbox 360/Kinect on Black Friday, I have to say, it is cool. BUT... It is slow. it isn't as simple as "touching" the air in the space you want it to be. You have to hover your hand in the area of the (rather large by necessity) button and hold it there for a few seconds while the sensor recognizes the hand, and to prevent any stray interaction, while a circle fills up around the "hand" to confirm. This works for gaming menus, but would be too slow to be useful on a computer. The dashboard update that came out Tuesday allows more voice interaction, which, in my limited use of it over the 40-48 hours it has been live, is pretty good, especially as a first "live" implementation. It isn't quite to the point of me being able to say "Play Fruit Ninja" from anywhere but more menu navigation in steps. "Xbox...Games...Play"



    Quote:
    Originally Posted by zoetmb View Post


    Because ideas like this are not supposed to be patentable, only the specific implementation of the idea. And while I will admit that the patent office violates that concept all the time (like permitting Amazon to patent "one-click", which Apple then had to license from Amazon), it's how patents are supposed to work. Otherwise, "information is presented on a screen" would be patentable.



    BINGO!! Not sure why this is so hard to grasp. ugh.
  • Reply 19 of 40
    gatorguygatorguy Posts: 24,176member
    Quote:
    Originally Posted by Doctor David View Post


    Who said it was "entirely new"? Are you making up quotes or just building a straw man?



    It's the very first line of the AI article.



    "Apple has shown interest in an entirely new way for users to interact. . ."
  • Reply 20 of 40
    Kinect-like? Too much avatars and games. We are creatives and this is a Mac. More like Theremin!
Sign In or Register to comment.