Apple exploring motion-tracking Mac OS X user interface
Patent filings from Apple continue to explore concepts for new interface designs and techniques that may or may not make turn up in future versions of Mac OS X, such as a new filing that outlines a motion-tracking interface in which body movements alone can be used to select windows and manipulate objects on the screen.
Motion Tracking User Interface
In the 20-page filing published this week, Apple notes that input devices for computer systems commonly include a mouse, a keyboard, a stylus, a track ball, and so forth, in which each of those input devices requires a user's spare hand to operate. But in some cases, it may be more efficient for both of the user's hands to remain free to type, which inconveniently interrupts manipulation of other input devices.
The Mac maker's proposed solution to this problem is fairly straightforward conceptually, but may prove incredibly challenging to implement with precision. Specifically, it calls for a Mac's built-in iSight camera to continuously monitor a user's body motions, which could then be translated into commands for selecting windows or user interface elements, manipulating 3D objects, and shifting focus from one object to another on a computer display.
Selection of a Target-Zone
In order to increase the accuracy of the iSight's motion tracking, Apple explains that users would be able to calibrate the technology by selecting a "target zone" on the body part or object that would be put under constant surveillance. For instance, if the body part was the users head, the focal point could be the portion of the user's face surrounding his or her eyes. Given an object, such as a pen, the focal point could instead be the pen's cap.
However, "in some implementations, selecting a target zone to be tracked by physical clicking is not necessary," Apple notes. "For example, optical flow information can be used to automatically detect a moving object without additional effort by a user to select a target zone to be tracked."
Focus Transition of a Display
The illustration below presents an example of an iSight following a user's face in which a pointer is displayed on the computer screen to correspond to the window in which the user is focusing his or her attention.
"[The iSight] can detect the movement of the human head or other objects, and in turn place the pointer on the window intended by the user to be focused on," Apple says. "For example, starting from a point in [one] window, if the human head moves from upper left to lower right, the pointer will move correspondingly from upper left to lower right and then bring [a new] window [into] to focus. When the [new] window is in focus, the [previous] window will be automatically raised to the top of all the windows such that the user can clearly see the current focus of the display."
After the user selects one of the multiple windows shown on the display, the he or she could start to hover the pointer within the selected window by slowly moving his or her head over relatively short distances.
If the window show on the computer screen includes an editor or a browser, a user could also scroll a digital file presented by the editor or the browser in different directions such as up, down, left, and right simply by moving his head or the other object.
"For example, say there is a five-page file presented by a word processor on a display, and only the first page is shown within the workspace of the word processor. If the user would like to see the last page of the file, he could simply move his head down to scroll the digital file down to the last page," Apple says. :Optionally, he could flip the digital file page-by-page by slightly moving his head down over a relatively short distance."
Manipulation of a Graphical Icon
The next example portrays a 3D icon object being manipulated by a user's head movements. For example, the user could turn his or her head to the right to make the teapot turn to the right, or turn to the left to make the teapot turn to the left. Thus, in a three-dimensional display environment, the head or object motion being tracked by the iSight camera, can cause a graphical object to move in six degrees of freedom, including up, down, left, right, backward, forward, rotations, etc.
"In some implementations, the graphical icon can serve as an avatar representing a user. A user could control eye positions of an avatar through head motions," Apple notes. "For example, when the user looks right, his head motion will trigger the avatar to look right. When the user looks left, his head motion will trigger the avatar look left. In other implementations, the avatar could simulate movements that the user makes, other than rotating or linear moving."
Manipulation of the Graphical Icon and Focus Transition of the Display
As shown in the example below, the user is moving the teapot by moving his head in a linear path from right to left. At the same time, the user is arbitrarily manipulating the teapot, like rotating it counterclockwise. This way the user can switch the current focus of the display between the two windows, as well as manipulate the teapot at the same time.
Additional Concepts
Apple goes on to note in its filing several additional concepts for manipulating objects and windows based on the same aforementioned principles. For example, it notes that the concept could be applied to computer setups featuring multiple displays. Objects could also be displayed from different angles based on a user's perspective. And in some cases, the aspects of a three-dimensional space can look closer or farer depending on the user's distance to the aspects shown on the computer screen.
The filing is credited to Apple employee Kevin Quennesson.
Motion Tracking User Interface
In the 20-page filing published this week, Apple notes that input devices for computer systems commonly include a mouse, a keyboard, a stylus, a track ball, and so forth, in which each of those input devices requires a user's spare hand to operate. But in some cases, it may be more efficient for both of the user's hands to remain free to type, which inconveniently interrupts manipulation of other input devices.
The Mac maker's proposed solution to this problem is fairly straightforward conceptually, but may prove incredibly challenging to implement with precision. Specifically, it calls for a Mac's built-in iSight camera to continuously monitor a user's body motions, which could then be translated into commands for selecting windows or user interface elements, manipulating 3D objects, and shifting focus from one object to another on a computer display.
Selection of a Target-Zone
In order to increase the accuracy of the iSight's motion tracking, Apple explains that users would be able to calibrate the technology by selecting a "target zone" on the body part or object that would be put under constant surveillance. For instance, if the body part was the users head, the focal point could be the portion of the user's face surrounding his or her eyes. Given an object, such as a pen, the focal point could instead be the pen's cap.
However, "in some implementations, selecting a target zone to be tracked by physical clicking is not necessary," Apple notes. "For example, optical flow information can be used to automatically detect a moving object without additional effort by a user to select a target zone to be tracked."
Focus Transition of a Display
The illustration below presents an example of an iSight following a user's face in which a pointer is displayed on the computer screen to correspond to the window in which the user is focusing his or her attention.
"[The iSight] can detect the movement of the human head or other objects, and in turn place the pointer on the window intended by the user to be focused on," Apple says. "For example, starting from a point in [one] window, if the human head moves from upper left to lower right, the pointer will move correspondingly from upper left to lower right and then bring [a new] window [into] to focus. When the [new] window is in focus, the [previous] window will be automatically raised to the top of all the windows such that the user can clearly see the current focus of the display."
After the user selects one of the multiple windows shown on the display, the he or she could start to hover the pointer within the selected window by slowly moving his or her head over relatively short distances.
If the window show on the computer screen includes an editor or a browser, a user could also scroll a digital file presented by the editor or the browser in different directions such as up, down, left, and right simply by moving his head or the other object.
"For example, say there is a five-page file presented by a word processor on a display, and only the first page is shown within the workspace of the word processor. If the user would like to see the last page of the file, he could simply move his head down to scroll the digital file down to the last page," Apple says. :Optionally, he could flip the digital file page-by-page by slightly moving his head down over a relatively short distance."
Manipulation of a Graphical Icon
The next example portrays a 3D icon object being manipulated by a user's head movements. For example, the user could turn his or her head to the right to make the teapot turn to the right, or turn to the left to make the teapot turn to the left. Thus, in a three-dimensional display environment, the head or object motion being tracked by the iSight camera, can cause a graphical object to move in six degrees of freedom, including up, down, left, right, backward, forward, rotations, etc.
"In some implementations, the graphical icon can serve as an avatar representing a user. A user could control eye positions of an avatar through head motions," Apple notes. "For example, when the user looks right, his head motion will trigger the avatar to look right. When the user looks left, his head motion will trigger the avatar look left. In other implementations, the avatar could simulate movements that the user makes, other than rotating or linear moving."
Manipulation of the Graphical Icon and Focus Transition of the Display
As shown in the example below, the user is moving the teapot by moving his head in a linear path from right to left. At the same time, the user is arbitrarily manipulating the teapot, like rotating it counterclockwise. This way the user can switch the current focus of the display between the two windows, as well as manipulate the teapot at the same time.
Additional Concepts
Apple goes on to note in its filing several additional concepts for manipulating objects and windows based on the same aforementioned principles. For example, it notes that the concept could be applied to computer setups featuring multiple displays. Objects could also be displayed from different angles based on a user's perspective. And in some cases, the aspects of a three-dimensional space can look closer or farer depending on the user's distance to the aspects shown on the computer screen.
The filing is credited to Apple employee Kevin Quennesson.
Comments
he could simply move his head down to scroll the digital file down to the last page
You have got to be joking.
You have got to be joking.
WHAT THIS MEANS IS THAT APPLE WITH HAVE
SOME FANTASTIC FUTURE GAMES . Thinking it thru is now dead. When the action gets heavy you only move your head to move the head and body on your game play screen. Think of it as wi for the head ..
Now i see why apple puts 2 chips in one computer .THese 2 gpu chips will work together to support the 3d gaming and the new head motion system. HALO should be first
Whew. APPLE ROCKS
9
Sounds very interesting.
Anyhow, what I would hope is that Apple would do something USEFUL with this idea. The only place I've seen multi-touch done well so far is on the iPhone devices. I'd LOVE to see multi touch come to Final Cut! Imagine manipulating your footage on the timeline by touching it! Even though keyboard is faster now, imagine how fast and interactive editing could get it we could touch it. Combine it with this tech, and you have something there. And same goes for image editing, music composition, all the arts.
But until then... this is just another toy. Nothing innovative (yet) as its been done before, and the basic idea has been around for quite some time.
Leave it to me to ignore the potential fun we could have in games with this LOL
:l: WHAT THIS MEANS IS THAT APPLE WITH HAVE
SOME FANTASTIC FUTURE GAMES . Thinking it thru is now dead. When the action gets heavy you only move your head to move the head and body on your game play screen. Think of it as wi for the head ..
Now i see why apple puts 2 chips in one computer .THese 2 gpu chips will work together to support the 3d gaming and the new head motion system. HALO should be first
Whew. APPLE ROCKS
9
You mean, like Micro$oft's Project Natal, or should I say, PROJECT LAME GIMMICK?
And by turning your head, you would get your vision AWAY FROM THE ACTION!
Let's learn from HP's touchscreen computer: it failed not only because of a poor software application, but because they apparently forgot gorilla's arm...
This kind of exaggerated motion sensing is best left for those lame casual Wii titles.
I'm afraid Sony's more discreet recent approach to motion sensing is the only way we'll get good titles that don't star an Italian plumber...
We'll see how Apple handles motion sensing...
http://www.eyetwig.com/151141
You mean, like Micro$oft's Project Natal, or should I say, PROJECT LAME GIMMICK?
And by turning your head, you would get your vision AWAY FROM THE ACTION!
Let's learn from HP's touchscreen computer: it failed not only because of a poor software application, but because they apparently
This kind of exaggerated motion sensing is best left for those lame casual Wii titles.
I'm afraid Sony's more discreet recent approach to motion sensing is the only way we'll get good titles that don't star an Italian plumber...
We'll see how Apple handles motion sensing...
WOW I didn't know . I hope apple does what you say. DISCRETE motion.
Personally, I'd hate for my iMac to begin responding to my head movement or facial patterns. That could lead to some very strange results.
But I can see a place for an 'almost touch' interface, particularly in work situations where a touch screen would have to be cleaned far too often. It would have some sort of scanning scheme mounted on the sides of a display that could spot fingers held close to the screen, say an inch or so away. It'd be multi-almost-touch and clever enough to understand gestures.
Dragging would move a window around. Making an X with a finger would close a window. Pinching in or out would decrease or increase the size of an image. Poking in and out with a finger would make a selection. In short, it would do most things now done with with an iPhone screen but without quite touching. With a little practice, users would learn where the sensitive area is and not leave greasy marks.
And it'd be a lot like the screens in "Minority Report" without the most annoying feature of the movie's imaginative interface. Having something behind the screen showing through isn't a good idea. I know I'd get a headache if I had to spend much time making a near versus far distinction. It'd be like reading a book with transparent paper. This would be much the same thing but with an opaque screen directly in front of what's being manipulated. That I could like and I suspect kids would love it.
--Mike Perry, Seattle
DOH! I threw my back out trying to gesture through a 500 page PDF...AGGGGHHHHHH, it hurts!!
first page is shown within the workspace of the word processor. If the user would like to see the last page of the file, he could simply move his head down to scroll the digital file down to the last page," Apple says. :Optionally, he could flip the digital file page-by-page by slightly moving his head down over a relatively short distance."
Their project Natal....if they add support for it to windows 7, it will be game over for Apple then......at least when it comes to the desktop with minority report type of controls. There will be no way for Apple to implement it for the desktop without looking like they are playing catch up.
Its already game over for Apple when it comes to the appletv. The natal is going to transform the xbox into truly a livingroom computer beyond simply a game system.
That, and the fact that the xbox will let you download 1080p....yes 1080....P HD movies.....the combo will be deadly to appletv. Oh, and the xbox will allow you to download games as well.
Apple will never be able to catch up in the living room unless they finally pull out all the stops and update the hell out of appletv.
I'm waiting on microsoft to announce windows 7 support for the natal. If they do then I'm definately not getting a mac.
I will however be getting the apple tablet so in a way I guess I will be getting a mac.
Or, what about Spaces? Swipe with the hand, and it moves to that Space.
Personally, I see a lot of potential with technology like this.
Its already game over for Apple when it comes to the appletv. The natal is going to transform the xbox into truly a livingroom computer beyond simply a game system.
That, and the fact that the xbox will let you download 1080p....yes 1080....P HD movies.....the combo will be deadly to appletv. Oh, and the xbox will allow you to download games as well.
Apple will never be able to catch up in the living room unless they finally pull out all the stops and update the hell out of appletv.
I'm waiting on microsoft to announce windows 7 support for the natal. If they do then I'm definately not getting a mac.
I will however be getting the apple tablet so in a way I guess I will be getting a mac.
I love Apple products, and despise Microsoft business tacticts, but this is right on the mark. Natal is ridiculously impressive and may well be for the living room what multitouch is for the phone.
http://cameramouse.org/index.html