or Connect
AppleInsider › Forums › Software › Mac OS X › Apple exploring motion-tracking Mac OS X user interface
New Posts  All Forums:Forum Nav:

Apple exploring motion-tracking Mac OS X user interface

post #1 of 25
Thread Starter 
Patent filings from Apple continue to explore concepts for new interface designs and techniques that may or may not make turn up in future versions of Mac OS X, such as a new filing that outlines a motion-tracking interface in which body movements alone can be used to select windows and manipulate objects on the screen.

Motion Tracking User Interface

In the 20-page filing published this week, Apple notes that input devices for computer systems commonly include a mouse, a keyboard, a stylus, a track ball, and so forth, in which each of those input devices requires a user's spare hand to operate. But in some cases, it may be more efficient for both of the user's hands to remain free to type, which inconveniently interrupts manipulation of other input devices.

The Mac maker's proposed solution to this problem is fairly straightforward conceptually, but may prove incredibly challenging to implement with precision. Specifically, it calls for a Mac's built-in iSight camera to continuously monitor a user's body motions, which could then be translated into commands for selecting windows or user interface elements, manipulating 3D objects, and shifting focus from one object to another on a computer display.

Selection of a Target-Zone

In order to increase the accuracy of the iSight's motion tracking, Apple explains that users would be able to calibrate the technology by selecting a "target zone" on the body part or object that would be put under constant surveillance. For instance, if the body part was the users head, the focal point could be the portion of the user's face surrounding his or her eyes. Given an object, such as a pen, the focal point could instead be the pen's cap.







However, "in some implementations, selecting a target zone to be tracked by physical clicking is not necessary," Apple notes. "For example, optical flow information can be used to automatically detect a moving object without additional effort by a user to select a target zone to be tracked."

Focus Transition of a Display

The illustration below presents an example of an iSight following a user's face in which a pointer is displayed on the computer screen to correspond to the window in which the user is focusing his or her attention.

"[The iSight] can detect the movement of the human head or other objects, and in turn place the pointer on the window intended by the user to be focused on," Apple says. "For example, starting from a point in [one] window, if the human head moves from upper left to lower right, the pointer will move correspondingly from upper left to lower right and then bring [a new] window [into] to focus. When the [new] window is in focus, the [previous] window will be automatically raised to the top of all the windows such that the user can clearly see the current focus of the display."

After the user selects one of the multiple windows shown on the display, the he or she could start to hover the pointer within the selected window by slowly moving his or her head over relatively short distances.



If the window show on the computer screen includes an editor or a browser, a user could also scroll a digital file presented by the editor or the browser in different directions such as up, down, left, and right simply by moving his head or the other object.

"For example, say there is a five-page file presented by a word processor on a display, and only the first page is shown within the workspace of the word processor. If the user would like to see the last page of the file, he could simply move his head down to scroll the digital file down to the last page," Apple says. :Optionally, he could flip the digital file page-by-page by slightly moving his head down over a relatively short distance."

Manipulation of a Graphical Icon

The next example portrays a 3D icon object being manipulated by a user's head movements. For example, the user could turn his or her head to the right to make the teapot turn to the right, or turn to the left to make the teapot turn to the left. Thus, in a three-dimensional display environment, the head or object motion being tracked by the iSight camera, can cause a graphical object to move in six degrees of freedom, including up, down, left, right, backward, forward, rotations, etc.



"In some implementations, the graphical icon can serve as an avatar representing a user. A user could control eye positions of an avatar through head motions," Apple notes. "For example, when the user looks right, his head motion will trigger the avatar to look right. When the user looks left, his head motion will trigger the avatar look left. In other implementations, the avatar could simulate movements that the user makes, other than rotating or linear moving."

Manipulation of the Graphical Icon and Focus Transition of the Display

As shown in the example below, the user is moving the teapot by moving his head in a linear path from right to left. At the same time, the user is arbitrarily manipulating the teapot, like rotating it counterclockwise. This way the user can switch the current focus of the display between the two windows, as well as manipulate the teapot at the same time.



Additional Concepts

Apple goes on to note in its filing several additional concepts for manipulating objects and windows based on the same aforementioned principles. For example, it notes that the concept could be applied to computer setups featuring multiple displays. Objects could also be displayed from different angles based on a user's perspective. And in some cases, the aspects of a three-dimensional space can look closer or farer depending on the user's distance to the aspects shown on the computer screen.

The filing is credited to Apple employee Kevin Quennesson.
post #2 of 25
Quote:
Originally Posted by AppleInsider View Post

he could simply move his head down to scroll the digital file down to the last page

You have got to be joking.
post #3 of 25
Alright, which designer watched Minority report one time too many?
post #4 of 25
i see lots of sore necks in the future with this tech
post #5 of 25
:l:
Quote:
Originally Posted by retroneo View Post

You have got to be joking.

WHAT THIS MEANS IS THAT APPLE WITH HAVE

SOME FANTASTIC FUTURE GAMES . Thinking it thru is now dead. When the action gets heavy you only move your head to move the head and body on your game play screen. Think of it as wi for the head ..

Now i see why apple puts 2 chips in one computer .THese 2 gpu chips will work together to support the 3d gaming and the new head motion system. HALO should be first

Whew. APPLE ROCKS

9
whats in a name ? 
beatles
Reply
whats in a name ? 
beatles
Reply
post #6 of 25
Leave it to Apple . . .

Sounds very interesting.
post #7 of 25
This could be great! Instead of stupid old mouse and keyboard.... just get up and dance!
Believe nothing, no matter where you heard it, not even if I have said it, if it does not agree with your own reason and your own common sense.
Buddha
Reply
Believe nothing, no matter where you heard it, not even if I have said it, if it does not agree with your own reason and your own common sense.
Buddha
Reply
post #8 of 25
It seems like just another toy up their sleeve. People have thought about using body movements to control their computers before. PS2's EyeToy anyone? Or what about the QuickCam from the 1990s? Or what about (gasp) M$ and their new camera based X360 attachment?

Anyhow, what I would hope is that Apple would do something USEFUL with this idea. The only place I've seen multi-touch done well so far is on the iPhone devices. I'd LOVE to see multi touch come to Final Cut! Imagine manipulating your footage on the timeline by touching it! Even though keyboard is faster now, imagine how fast and interactive editing could get it we could touch it. Combine it with this tech, and you have something there. And same goes for image editing, music composition, all the arts.

But until then... this is just another toy. Nothing innovative (yet) as its been done before, and the basic idea has been around for quite some time.
Go Linux, Choose a Flavor!
"I aim to misbehave"
Reply
Go Linux, Choose a Flavor!
"I aim to misbehave"
Reply
post #9 of 25
This would be useful for the disabled. Currently, programs like Dragon Naturally Speaking obviously require you to speak to the computer, but this can be a security flaw when you go to login somewhere, or use personal information, all within a vicinity of listening ears.

Leave it to me to ignore the potential fun we could have in games with this LOL
post #10 of 25
Quote:
Originally Posted by brucep View Post

:l: WHAT THIS MEANS IS THAT APPLE WITH HAVE

SOME FANTASTIC FUTURE GAMES . Thinking it thru is now dead. When the action gets heavy you only move your head to move the head and body on your game play screen. Think of it as wi for the head ..

Now i see why apple puts 2 chips in one computer .THese 2 gpu chips will work together to support the 3d gaming and the new head motion system. HALO should be first

Whew. APPLE ROCKS

9

You mean, like Micro$oft's Project Natal, or should I say, PROJECT LAME GIMMICK?
And by turning your head, you would get your vision AWAY FROM THE ACTION!
Let's learn from HP's touchscreen computer: it failed not only because of a poor software application, but because they apparently forgot gorilla's arm...
This kind of exaggerated motion sensing is best left for those lame casual Wii titles.

I'm afraid Sony's more discreet recent approach to motion sensing is the only way we'll get good titles that don't star an Italian plumber...

We'll see how Apple handles motion sensing...

iPhone 4S 64GB, Black, soon to be sold in favor of a Nokia Lumia 920
Early 2010 MacBook Pro 2.4GHz, soon to be replaced with a Retina MacBook Pro, or an Asus U500

Reply

iPhone 4S 64GB, Black, soon to be sold in favor of a Nokia Lumia 920
Early 2010 MacBook Pro 2.4GHz, soon to be replaced with a Retina MacBook Pro, or an Asus U500

Reply
post #11 of 25
Reminds me a little or a lot of Microsoft's Project Natal and the PS3 motion wand.
post #12 of 25
Something similar exists now for the Mac, from personal use it seems more of a toy than anything useful, but it does exist. Not 100% like Apple speculates in their patent, but something to hold everyone over for what, a couple years?

http://www.eyetwig.com/151141
post #13 of 25
Quote:
Originally Posted by Lukeskymac View Post

You mean, like Micro$oft's Project Natal, or should I say, PROJECT LAME GIMMICK?
And by turning your head, you would get your vision AWAY FROM THE ACTION!
Let's learn from HP's touchscreen computer: it failed not only because of a poor software application, but because they apparently
This kind of exaggerated motion sensing is best left for those lame casual Wii titles.

I'm afraid Sony's more discreet recent approach to motion sensing is the only way we'll get good titles that don't star an Italian plumber...

We'll see how Apple handles motion sensing...

WOW I didn't know . I hope apple does what you say. DISCRETE motion.
whats in a name ? 
beatles
Reply
whats in a name ? 
beatles
Reply
post #14 of 25
This is a great article. Thanks for doing so much work to sort of what these patents mean. "Patentspeak" isn't easy to understand. I've never want to be a patent lawyer.

Personally, I'd hate for my iMac to begin responding to my head movement or facial patterns. That could lead to some very strange results.

But I can see a place for an 'almost touch' interface, particularly in work situations where a touch screen would have to be cleaned far too often. It would have some sort of scanning scheme mounted on the sides of a display that could spot fingers held close to the screen, say an inch or so away. It'd be multi-almost-touch and clever enough to understand gestures.

Dragging would move a window around. Making an X with a finger would close a window. Pinching in or out would decrease or increase the size of an image. Poking in and out with a finger would make a selection. In short, it would do most things now done with with an iPhone screen but without quite touching. With a little practice, users would learn where the sensitive area is and not leave greasy marks.

And it'd be a lot like the screens in "Minority Report" without the most annoying feature of the movie's imaginative interface. Having something behind the screen showing through isn't a good idea. I know I'd get a headache if I had to spend much time making a near versus far distinction. It'd be like reading a book with transparent paper. This would be much the same thing but with an opaque screen directly in front of what's being manipulated. That I could like and I suspect kids would love it.

--Mike Perry, Seattle
post #15 of 25
This all seems like a chiropractor's dream to me!

DOH! I threw my back out trying to gesture through a 500 page PDF...AGGGGHHHHHH, it hurts!!

Quote:
Originally Posted by AppleInsider View Post

first page is shown within the workspace of the word processor. If the user would like to see the last page of the file, he could simply move his head down to scroll the digital file down to the last page," Apple says. :Optionally, he could flip the digital file page-by-page by slightly moving his head down over a relatively short distance."
post #16 of 25
Microsoft has already beaten Apple to this..........for once.

Their project Natal....if they add support for it to windows 7, it will be game over for Apple then......at least when it comes to the desktop with minority report type of controls. There will be no way for Apple to implement it for the desktop without looking like they are playing catch up.

Its already game over for Apple when it comes to the appletv. The natal is going to transform the xbox into truly a livingroom computer beyond simply a game system.
That, and the fact that the xbox will let you download 1080p....yes 1080....P HD movies.....the combo will be deadly to appletv. Oh, and the xbox will allow you to download games as well.

Apple will never be able to catch up in the living room unless they finally pull out all the stops and update the hell out of appletv.
I'm waiting on microsoft to announce windows 7 support for the natal. If they do then I'm definately not getting a mac.
I will however be getting the apple tablet so in a way I guess I will be getting a mac.
post #17 of 25
I think that one of the most useful ways this could be implemented is tracking where the eyes are looking. Most people (myself included), they look straight at a windowed page, either playing a game, editing a photo, coding your website, or whatever. There has been several times where my eyes are already at the window I want to activate, and with larger screens, it can be very easy to lose your cursor.

Or, what about Spaces? Swipe with the hand, and it moves to that Space.

Personally, I see a lot of potential with technology like this.

-- Mike Eggleston
-- Mac Fanatic since 1984.
-- Proud Member of PETA: People Eating Tasty Animals
-- Wii #: 8913 3004 4519 2027

Reply

-- Mike Eggleston
-- Mac Fanatic since 1984.
-- Proud Member of PETA: People Eating Tasty Animals
-- Wii #: 8913 3004 4519 2027

Reply
post #18 of 25
Quote:
Originally Posted by Olternaut View Post

Its already game over for Apple when it comes to the appletv. The natal is going to transform the xbox into truly a livingroom computer beyond simply a game system.
That, and the fact that the xbox will let you download 1080p....yes 1080....P HD movies.....the combo will be deadly to appletv. Oh, and the xbox will allow you to download games as well.

Apple will never be able to catch up in the living room unless they finally pull out all the stops and update the hell out of appletv.
I'm waiting on microsoft to announce windows 7 support for the natal. If they do then I'm definately not getting a mac.
I will however be getting the apple tablet so in a way I guess I will be getting a mac.

I love Apple products, and despise Microsoft business tacticts, but this is right on the mark. Natal is ridiculously impressive and may well be for the living room what multitouch is for the phone.
post #19 of 25
Oh, brother. \
2011 13" 2.3 MBP, 2006 15" 2.16 MBP, iPhone 4, iPod Shuffle, AEBS, AppleTV2 with XBMC.
Reply
2011 13" 2.3 MBP, 2006 15" 2.16 MBP, iPhone 4, iPod Shuffle, AEBS, AppleTV2 with XBMC.
Reply
post #20 of 25
Open your eyes, people, we're talking about M$! Natal is just a slightly improved PS Eye that seems to be out of this world due to typical M$ marketing. Remember Cairo, Longhorn, Vienna, etc??

iPhone 4S 64GB, Black, soon to be sold in favor of a Nokia Lumia 920
Early 2010 MacBook Pro 2.4GHz, soon to be replaced with a Retina MacBook Pro, or an Asus U500

Reply

iPhone 4S 64GB, Black, soon to be sold in favor of a Nokia Lumia 920
Early 2010 MacBook Pro 2.4GHz, soon to be replaced with a Retina MacBook Pro, or an Asus U500

Reply
post #21 of 25
If you're on a PC you can give cameramouse a try. You can control the mouse with your eyes or face. It's really not that useful if you can use your hands, but motion gestures for things like moving to the next page might be cool... a guess, if not a little strange looking to witness.

http://cameramouse.org/index.html
"Education is what remains after one has forgotten what one has learned in school." -- Albert Einstein
Reply
"Education is what remains after one has forgotten what one has learned in school." -- Albert Einstein
Reply
post #22 of 25
This is not just directed at Apple.

Patents for ideas and processes should not be possible without some element of effort to bring about the result intended.
post #23 of 25
Ok, let's give credit where it is due. Microsoft demonstrated this years ago from their research department, not just some diagrams and ideas that they are trying to get Patents on. Also, the new Xbox technology they showed at G2 showed this working in real time on a real solution. Yes Apple is definitely a leading innovator but don't forget companies like Microsoft keep them going just as Apple puts pressure on Microsoft. So before you "applaud" Apple for submitting some ideas to patent, look at what others are DOING and SHOWING and DELIVERING.
post #24 of 25
This is really a great information......... Thanks for sharing......... Lets see what Apple does.....
[right]Sales tracking sales tracking software Sales lead tracking software[/right]
post #25 of 25
Pissoirs have been doing that for decades now...

We mean Apple no harm.

People are lovers, basically. -- Engadget livebloggers at the iPad mini event.

Reply

We mean Apple no harm.

People are lovers, basically. -- Engadget livebloggers at the iPad mini event.

Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: Mac OS X
AppleInsider › Forums › Software › Mac OS X › Apple exploring motion-tracking Mac OS X user interface