Apple exploring motion-tracking Mac OS X user interface

Posted:
in Mac OS X edited January 2014
Patent filings from Apple continue to explore concepts for new interface designs and techniques that may or may not make turn up in future versions of Mac OS X, such as a new filing that outlines a motion-tracking interface in which body movements alone can be used to select windows and manipulate objects on the screen.



Motion Tracking User Interface



In the 20-page filing published this week, Apple notes that input devices for computer systems commonly include a mouse, a keyboard, a stylus, a track ball, and so forth, in which each of those input devices requires a user's spare hand to operate. But in some cases, it may be more efficient for both of the user's hands to remain free to type, which inconveniently interrupts manipulation of other input devices.



The Mac maker's proposed solution to this problem is fairly straightforward conceptually, but may prove incredibly challenging to implement with precision. Specifically, it calls for a Mac's built-in iSight camera to continuously monitor a user's body motions, which could then be translated into commands for selecting windows or user interface elements, manipulating 3D objects, and shifting focus from one object to another on a computer display.



Selection of a Target-Zone



In order to increase the accuracy of the iSight's motion tracking, Apple explains that users would be able to calibrate the technology by selecting a "target zone" on the body part or object that would be put under constant surveillance. For instance, if the body part was the users head, the focal point could be the portion of the user's face surrounding his or her eyes. Given an object, such as a pen, the focal point could instead be the pen's cap.















However, "in some implementations, selecting a target zone to be tracked by physical clicking is not necessary," Apple notes. "For example, optical flow information can be used to automatically detect a moving object without additional effort by a user to select a target zone to be tracked."



Focus Transition of a Display



The illustration below presents an example of an iSight following a user's face in which a pointer is displayed on the computer screen to correspond to the window in which the user is focusing his or her attention.



"[The iSight] can detect the movement of the human head or other objects, and in turn place the pointer on the window intended by the user to be focused on," Apple says. "For example, starting from a point in [one] window, if the human head moves from upper left to lower right, the pointer will move correspondingly from upper left to lower right and then bring [a new] window [into] to focus. When the [new] window is in focus, the [previous] window will be automatically raised to the top of all the windows such that the user can clearly see the current focus of the display."



After the user selects one of the multiple windows shown on the display, the he or she could start to hover the pointer within the selected window by slowly moving his or her head over relatively short distances.







If the window show on the computer screen includes an editor or a browser, a user could also scroll a digital file presented by the editor or the browser in different directions such as up, down, left, and right simply by moving his head or the other object.



"For example, say there is a five-page file presented by a word processor on a display, and only the first page is shown within the workspace of the word processor. If the user would like to see the last page of the file, he could simply move his head down to scroll the digital file down to the last page," Apple says. :Optionally, he could flip the digital file page-by-page by slightly moving his head down over a relatively short distance."



Manipulation of a Graphical Icon



The next example portrays a 3D icon object being manipulated by a user's head movements. For example, the user could turn his or her head to the right to make the teapot turn to the right, or turn to the left to make the teapot turn to the left. Thus, in a three-dimensional display environment, the head or object motion being tracked by the iSight camera, can cause a graphical object to move in six degrees of freedom, including up, down, left, right, backward, forward, rotations, etc.







"In some implementations, the graphical icon can serve as an avatar representing a user. A user could control eye positions of an avatar through head motions," Apple notes. "For example, when the user looks right, his head motion will trigger the avatar to look right. When the user looks left, his head motion will trigger the avatar look left. In other implementations, the avatar could simulate movements that the user makes, other than rotating or linear moving."



Manipulation of the Graphical Icon and Focus Transition of the Display



As shown in the example below, the user is moving the teapot by moving his head in a linear path from right to left. At the same time, the user is arbitrarily manipulating the teapot, like rotating it counterclockwise. This way the user can switch the current focus of the display between the two windows, as well as manipulate the teapot at the same time.







Additional Concepts



Apple goes on to note in its filing several additional concepts for manipulating objects and windows based on the same aforementioned principles. For example, it notes that the concept could be applied to computer setups featuring multiple displays. Objects could also be displayed from different angles based on a user's perspective. And in some cases, the aspects of a three-dimensional space can look closer or farer depending on the user's distance to the aspects shown on the computer screen.



The filing is credited to Apple employee Kevin Quennesson.
«1

Comments

  • Reply 1 of 24
    retroneoretroneo Posts: 240member
    Quote:
    Originally Posted by AppleInsider View Post


    he could simply move his head down to scroll the digital file down to the last page



    You have got to be joking.
  • Reply 2 of 24
    sipadansipadan Posts: 107member
    Alright, which designer watched Minority report one time too many?
  • Reply 3 of 24
    hngfrhngfr Posts: 72member
    i see lots of sore necks in the future with this tech

  • Reply 4 of 24
    brucepbrucep Posts: 2,823member
    :l:
    Quote:
    Originally Posted by retroneo View Post


    You have got to be joking.



    WHAT THIS MEANS IS THAT APPLE WITH HAVE



    SOME FANTASTIC FUTURE GAMES . Thinking it thru is now dead. When the action gets heavy you only move your head to move the head and body on your game play screen. Think of it as wi for the head ..



    Now i see why apple puts 2 chips in one computer .THese 2 gpu chips will work together to support the 3d gaming and the new head motion system. HALO should be first



    Whew. APPLE ROCKS



    9
  • Reply 5 of 24
    quadra 610quadra 610 Posts: 6,679member
    Leave it to Apple . . .



    Sounds very interesting.
  • Reply 6 of 24
    allblueallblue Posts: 393member
    This could be great! Instead of stupid old mouse and keyboard.... just get up and dance!
  • Reply 7 of 24
    It seems like just another toy up their sleeve. People have thought about using body movements to control their computers before. PS2's EyeToy anyone? Or what about the QuickCam from the 1990s? Or what about (gasp) M$ and their new camera based X360 attachment?



    Anyhow, what I would hope is that Apple would do something USEFUL with this idea. The only place I've seen multi-touch done well so far is on the iPhone devices. I'd LOVE to see multi touch come to Final Cut! Imagine manipulating your footage on the timeline by touching it! Even though keyboard is faster now, imagine how fast and interactive editing could get it we could touch it. Combine it with this tech, and you have something there. And same goes for image editing, music composition, all the arts.



    But until then... this is just another toy. Nothing innovative (yet) as its been done before, and the basic idea has been around for quite some time.
  • Reply 8 of 24
    chronsterchronster Posts: 1,894member
    This would be useful for the disabled. Currently, programs like Dragon Naturally Speaking obviously require you to speak to the computer, but this can be a security flaw when you go to login somewhere, or use personal information, all within a vicinity of listening ears.



    Leave it to me to ignore the potential fun we could have in games with this LOL
  • Reply 9 of 24
    Quote:
    Originally Posted by brucep View Post


    :l: WHAT THIS MEANS IS THAT APPLE WITH HAVE



    SOME FANTASTIC FUTURE GAMES . Thinking it thru is now dead. When the action gets heavy you only move your head to move the head and body on your game play screen. Think of it as wi for the head ..



    Now i see why apple puts 2 chips in one computer .THese 2 gpu chips will work together to support the 3d gaming and the new head motion system. HALO should be first



    Whew. APPLE ROCKS



    9



    You mean, like Micro$oft's Project Natal, or should I say, PROJECT LAME GIMMICK?

    And by turning your head, you would get your vision AWAY FROM THE ACTION!

    Let's learn from HP's touchscreen computer: it failed not only because of a poor software application, but because they apparently forgot gorilla's arm...

    This kind of exaggerated motion sensing is best left for those lame casual Wii titles.



    I'm afraid Sony's more discreet recent approach to motion sensing is the only way we'll get good titles that don't star an Italian plumber...



    We'll see how Apple handles motion sensing...
  • Reply 10 of 24
    unicronunicron Posts: 146member
    Reminds me a little or a lot of Microsoft's Project Natal and the PS3 motion wand.
  • Reply 11 of 24
    Something similar exists now for the Mac, from personal use it seems more of a toy than anything useful, but it does exist. Not 100% like Apple speculates in their patent, but something to hold everyone over for what, a couple years?



    http://www.eyetwig.com/151141
  • Reply 12 of 24
    brucepbrucep Posts: 2,823member
    Quote:
    Originally Posted by Lukeskymac View Post


    You mean, like Micro$oft's Project Natal, or should I say, PROJECT LAME GIMMICK?

    And by turning your head, you would get your vision AWAY FROM THE ACTION!

    Let's learn from HP's touchscreen computer: it failed not only because of a poor software application, but because they apparently

    This kind of exaggerated motion sensing is best left for those lame casual Wii titles.



    I'm afraid Sony's more discreet recent approach to motion sensing is the only way we'll get good titles that don't star an Italian plumber...



    We'll see how Apple handles motion sensing...



    WOW I didn't know . I hope apple does what you say. DISCRETE motion.
  • Reply 13 of 24
    inklinginkling Posts: 731member
    This is a great article. Thanks for doing so much work to sort of what these patents mean. "Patentspeak" isn't easy to understand. I've never want to be a patent lawyer.



    Personally, I'd hate for my iMac to begin responding to my head movement or facial patterns. That could lead to some very strange results.



    But I can see a place for an 'almost touch' interface, particularly in work situations where a touch screen would have to be cleaned far too often. It would have some sort of scanning scheme mounted on the sides of a display that could spot fingers held close to the screen, say an inch or so away. It'd be multi-almost-touch and clever enough to understand gestures.



    Dragging would move a window around. Making an X with a finger would close a window. Pinching in or out would decrease or increase the size of an image. Poking in and out with a finger would make a selection. In short, it would do most things now done with with an iPhone screen but without quite touching. With a little practice, users would learn where the sensitive area is and not leave greasy marks.



    And it'd be a lot like the screens in "Minority Report" without the most annoying feature of the movie's imaginative interface. Having something behind the screen showing through isn't a good idea. I know I'd get a headache if I had to spend much time making a near versus far distinction. It'd be like reading a book with transparent paper. This would be much the same thing but with an opaque screen directly in front of what's being manipulated. That I could like and I suspect kids would love it.



    --Mike Perry, Seattle
  • Reply 14 of 24
    This all seems like a chiropractor's dream to me!



    DOH! I threw my back out trying to gesture through a 500 page PDF...AGGGGHHHHHH, it hurts!!



    Quote:
    Originally Posted by AppleInsider View Post


    first page is shown within the workspace of the word processor. If the user would like to see the last page of the file, he could simply move his head down to scroll the digital file down to the last page," Apple says. :Optionally, he could flip the digital file page-by-page by slightly moving his head down over a relatively short distance."



  • Reply 15 of 24
    olternautolternaut Posts: 1,376member
    Microsoft has already beaten Apple to this..........for once.



    Their project Natal....if they add support for it to windows 7, it will be game over for Apple then......at least when it comes to the desktop with minority report type of controls. There will be no way for Apple to implement it for the desktop without looking like they are playing catch up.



    Its already game over for Apple when it comes to the appletv. The natal is going to transform the xbox into truly a livingroom computer beyond simply a game system.

    That, and the fact that the xbox will let you download 1080p....yes 1080....P HD movies.....the combo will be deadly to appletv. Oh, and the xbox will allow you to download games as well.



    Apple will never be able to catch up in the living room unless they finally pull out all the stops and update the hell out of appletv.

    I'm waiting on microsoft to announce windows 7 support for the natal. If they do then I'm definately not getting a mac.

    I will however be getting the apple tablet so in a way I guess I will be getting a mac.
  • Reply 16 of 24
    I think that one of the most useful ways this could be implemented is tracking where the eyes are looking. Most people (myself included), they look straight at a windowed page, either playing a game, editing a photo, coding your website, or whatever. There has been several times where my eyes are already at the window I want to activate, and with larger screens, it can be very easy to lose your cursor.



    Or, what about Spaces? Swipe with the hand, and it moves to that Space.



    Personally, I see a lot of potential with technology like this.
  • Reply 17 of 24
    animaniacanimaniac Posts: 122member
    Quote:
    Originally Posted by Olternaut View Post


    Its already game over for Apple when it comes to the appletv. The natal is going to transform the xbox into truly a livingroom computer beyond simply a game system.

    That, and the fact that the xbox will let you download 1080p....yes 1080....P HD movies.....the combo will be deadly to appletv. Oh, and the xbox will allow you to download games as well.



    Apple will never be able to catch up in the living room unless they finally pull out all the stops and update the hell out of appletv.

    I'm waiting on microsoft to announce windows 7 support for the natal. If they do then I'm definately not getting a mac.

    I will however be getting the apple tablet so in a way I guess I will be getting a mac.



    I love Apple products, and despise Microsoft business tacticts, but this is right on the mark. Natal is ridiculously impressive and may well be for the living room what multitouch is for the phone.
  • Reply 18 of 24
    8corewhore8corewhore Posts: 833member
    Oh, brother. \
  • Reply 19 of 24
    lukeskymaclukeskymac Posts: 506member
    Open your eyes, people, we're talking about M$! Natal is just a slightly improved PS Eye that seems to be out of this world due to typical M$ marketing. Remember Cairo, Longhorn, Vienna, etc??
  • Reply 20 of 24
    dogcowdogcow Posts: 713member
    If you're on a PC you can give cameramouse a try. You can control the mouse with your eyes or face. It's really not that useful if you can use your hands, but motion gestures for things like moving to the next page might be cool... a guess, if not a little strange looking to witness.



    http://cameramouse.org/index.html
Sign In or Register to comment.