Apple looks to take multi-touch beyond the touch-screen

[AppleInsider]

With its competitors struggling to catch up with multi-touch technology introduced last year as part of the iPhone, Apple is already conceptualizing new versions of the technology that would fuse a variety of secondary inputs with today's touch-based gestures to produce more efficient data input operations.

A new 30-page patent filing by Wayne Westerman and John Elias, co-founders of the Fingerworks firm acquired by Apple during the development of the original iPhone, details a handful of these newly proposed inputs under the title "Multi-Touch Data Fusion."

The pair of engineers note that while the fingertip chording and movement data generated by today's multi-touch input devices can provide a strong set of user control, fusing additional information from other sensing modalities can significantly enhance the interpretative abilities of a device or significantly improve its overall ease of use.

Among the secondary input means outlined in the filing (and detailed below) are voice fusion, finger identification fusion, gaze vector fusion, biometrics fusion, and facial expression fusion.

Voice Fusion

In the case of Voice Fusion, it's proposed that voice input be applied to actions on a multi-touch device that are poorly served by manual input while manual input handles tasks poorly served by voice.

For example, the filing presents the scenario where modifications to a mechanical drawing require resizing, rotation, and color change. Using voice commands to resize and rotate objects in the drawing would prove problematic because a verbal description of the intended size and rotation would be difficult to express. Therefor, those operations are best suited for manipulation by manual multi-touch gestures.



On the other hand, "using multi-touch to select a color is typically less efficient than using voice because the color has to be selected by traversing a list," Westerman explained. "Alternatively or additionally, voice input may be used to insert text in the object."

Finger Identification Fusion

Today's multi-touch touch sensors are efficient at detecting the presence of human fingers, but they do so with ambiguity as to which specific fingers are making the contacts. While that limitation doesn't pose a problem for existing applications, there are some examples where precise finger identification is imperative.

"Finger painting, where each finger has an assigned color, stroke, or other characteristic, is a simple example of an application that would be significantly enhanced compared to the state-of-the-art by using finger identification with multi-touch data fusion," Westerman wrote. "For example, if the index finger of the left hand is assigned the color red and the other fingers are assigned different colors the application must be able to determine when the index finger of the left hand is in contact with the surface in order to paint red. Conversely, the application must be able to determine when the red-assigned finger is not in contact with the surface. The fusion of finger identification data with multi-touch movement data allows the application to function without error."



To help identify individual fingers, the engineer proposes the user of a digital video camera, like Apple's built-in iSights, to provide a view over the multi-touch surface. The camera data would determine where the fingers of each hand are relative to the multi-touch XY coordinates, while the multi-touch sensor determines when the fingers make contact with the multi-touch surface.

Gaze Vector Fusion

Similarly, iSight cameras could also serve to record gaze vector data, where operations on a computer screen are partially determined by the direction in which a user directs his eyes or head position.



For example, if the user wishes to bring forward a window in the lower left corner of a screen, which is currently underneath two other windows, the user would direct his gaze to the window of interest and then tap a specific chord on the multi-touch surface.

Biometrics Fusion

The filing also suggests the use of biometric input, or input that is determined by hand size, fingerprint input, body temperature, heart rate, skin impedance, and pupil size.

"Typical applications that might benefit from the fusion of biometric data with multi-touch movement data would include games, security, and fitness related activities," Westerman wrote. "Hand characteristics such as size, shape, and general morphology can be used to identify an individual for the purpose of allowing access to secured areas, including computer systems. While hand characteristics alone would not provide a sufficient level of identity verification, it could be the first door through which a user must pass before other security measures are applied."

Facial Expression Fusion

A final means of secondary input outlined in Westerman's filing, which seems years off, would again employ a digital video camera like an iSight, to determine operations based on a user's facial expression and signs of frustration.



For example, he notes that a novice user may experience frustration from time to time when learning how to perform some task with an electronic device. "Say that the user is trying to scroll through a document using a two-finger vertical movement (gesture). Scrolling, however, is not working for him because he is unknowingly touching the surface with three fingers instead of the required two," he wrote. "He becomes frustrated with the 'failure' of the device. However, in this case, the system recognizes the frustration and upon analyzing the multi-touch movement data concludes he is trying to scroll with three fingers. At this point, the device could bring the extra-finger problem to the attention of the user or it could decide to ignore the extra finger and commence scrolling. Subsequent emotional data via facial recognition would confirm to the system that the correct remedial action was taken."

The filing by Westerman and Elias is dated Dec. 27, 2007.

[ View this article at AppleInsider.com ]

[dr_lha]

I can see it now, in Office 2020:

Clippy: You look frustrated, can I help you?
User: YArrrgghhhhhh!!!!!

[eAi]

What about hearing the user shout at the computer? Detecting tone of voice etc

[wbrasington]

Finally!
A way to do cut/copy/paste on the iPhone.
The user uses voice commands and clearly prnounces..... "Control - A, Control - VEE"

Oh, so simple.....

[GordonComstock]

I know we all want to be touchy-feely with the iPhone, but how about a simple external modifier key? Put it on the opposite side of the iPhone from the volume buttons. Heck make it two keys like Shift and Command!

There's no efficient equivalent of the "click and hold" feature of a mouse on the iPhone. When copy and paste arrives, I'm betting that it's no picnic.

99.9% of the time, when operating an iPhone, you're holding it in one hand and tapping with the other. Give that other hand a conveniently placed button! Apple's already realized that the iTouch needs external buttons --it's not always convenient to have EVERYTHING handled by onscreen touch control. I don't think the iPod experience on the iPhone is as good as a click wheel endowed iPod.

gc

[SpamSandwich]

All very interesting, but hopefully patents applied for that they never intend to use in real products. The only recent idea I've been taken with is the idea of a multi-touch mouse or a keyboard that adds some kind of mulit-touch and yet retains the feel of a keyboard. At least then the fingers can remain in a good position to continue to do work. Any time the hands even leave the keyboard to go to the mouse is wasted movement.

[solipsism]

Originally Posted by wbrasington

Finally!
A way to do cut/copy/paste on the iPhone.
The user uses voice commands and clearly prnounces..... "Control - A, Control - VEE"

Oh, so simple.....

That would be horrible. Think of some of the horrible key combinations. I'm hoping that copy and paste commands are done very similar to how the maker of MagicPad envisioned it.

• http://magicpad.proximi.com/video.php

[macinthe408]

Just watch Tom Cruise in "Minority Report" and you get a sense of where this is going. Seriously.

If you thought that what you saw was not awesome, raise your hand. Not a single hand goes up!

[Lafe]

Smile, and your iMac smiles with you.

[macosxp]

Some of that is kind of creepy, a computer that senses so much about you...

And what if you want your son to do something on the computer for you? "My dad says it's okay! I even have his authorization code!" iMac: "I'm sorry, I can't let you do that Johnny. Your hands are too little."

All I want is to tell my iMac "define disestablishment" and have Dictionary pop up with the definition.

As it stands, speech recognition sucks. I tell the computer "Open Pages" and it says "Opening iTunes" lol. And it hardly ever responds when I tell it to move a piece in Chess.

[FuturePastNow]

I'm dreaming of the Star Trek computer that you can just speak naturally to, and that doesn't have to be "trained" for a specific person. But I don't think voice recognition is anywhere near that level yet.

[wbrasington]

Originally Posted by FuturePastNow

I'm dreaming of the Star Trek computer that you can just speak naturally to, and that doesn't have to be "trained" for a specific person. But I don't think voice recognition is anywhere near that level yet.

Actually, a lot of very good stuff doesn't need training and that is getting better all the time.
Just try to call Vonage for support and you get to use an untrained voice recognition system that does very good. (but is still as annoying as all menu driven call center systems)

Check out the move Bladerunner.
Your computer's going to watch your eye and decide if you're an organism that can be trusted by the collective. (sorry too many references there)

[8CoreWhore]

"Dave, although you took thorough precautions in the pod against my hearing you, I could see your lips move." "

[solipsism]

Originally Posted by macosxp

All I want is to tell my iMac "define disestablishment" and have Dictionary pop up with the definition.

You can't do that but there is an easier way to get access to the Dictionary for very quick referencing. The current use of the mini-dictionary within most apps requires a complex use of keys but if you map it to an F-key it makes it a toggle key so you don't have to hold that combo down and you can now move around your desktop freely with the mouse and any word it crosses over will pop up instantly with a definition.

[wbrasington]

Originally Posted by 8CoreWhore

"Dave, although you took thorough precautions in the pod against my hearing you, I could see your lips move." "

Very nice....

[trevorlsciact]

Originally Posted by solipsism

You can't do that but there is an easier way to get access to the Dictionary for very quick referencing. The current use of the mini-dictionary within most apps requires a complex use of keys but if you map it to an F-key it makes it a toggle key so you don't have to hold that combo down and you can now move around your desktop freely with the mouse and any word it crosses over will pop up instantly with a definition.

I think that should be how it is by default tho, with an easier way to change it.

[iVlad]

I just love how much Apple in inventing and patent everything. They will always be leader as long as they have the best people working for them.

[solipsism]

Originally Posted by trevorlsciact

I think that should be how it is by default tho, with an easier way to change it.

Me too. It's one of the first thing I change on every Mac I set up. I think they need to include one of those nice little printed icons on the PF key to denote what it is, too. It's just so dang useful!

[SpamSandwich]

Originally Posted by macinthe408

Just watch Tom Cruise in "Minority Report" and you get a sense of where this is going. Seriously.

If you thought that what you saw was not awesome, raise your hand. Not a single hand goes up!

I'm raising my hand. A fully gesture driven computer would be a huge waste of movement and energy. It's inefficient. The Minority Report interface looked the way it did because of Jeff Hahn's work and because Tom Cruise was supposed to look like a conductor in complete command of his craft.

[Buck]

Actually this IS exciting. I think this is where the future of user interfaces lies. Computers that control our every move, our minute emotions and sense our heartbeat and the slightest tremors. Yep, that's the future and I'm glad it's finally here.

[solipsism]

Originally Posted by SpamSandwich

I'm raising my hand. A fully gesture driven computer would be a huge waste of movement and energy. It's inefficient. The Minority Report interface looked the way it did because of Jeff Hahn's work and because Tom Cruise was supposed to look like a conductor in complete command of his craft.

I'm right there with you. The move to new tech isn't about being cool to look at in a movie, it's about feeling natural. We want to use less effort and get more for the result and moving windows around with wild arm gestures and fancy hand twists isn't going to cut it except in Hollywwood.

Originally Posted by Buck

Actually this IS exciting. I think this is where the future of user interfaces lies. Computers that control our every move, our minute emotions and sense our heartbeat and the slightest tremors. Yep, that's the future and I'm glad it's finally here.

What you are describing is a long way off. There has to be baby steps for each transition. As pe a recent RD article, I think that a mutli-touch trackpad (note that they already are multitouch on the MBP and MBA) with a visual output is the next step.

[SpamSandwich]

Originally Posted by solipsism

What you are describing is a long way off. There has to be baby steps for each transition. As pe a recent RD article, I think that a mutli-touch trackpad (note that they already are multitouch on the MBP and MBA) with a visual output is the next step.

I could really see some functional value in the trackpad idea, perhaps as an alternative to the mouse, but "clicking" on a trackpad is a poor alternative to a physical switch/button.

[solipsism]

Originally Posted by SpamSandwich

I could really see some functional value in the trackpad idea, perhaps as an alternative to the mouse, but "clicking" on a trackpad is a poor alternative to a physical switch/button.

I'm not sure I get what you are saying. I don't see the physical mouse button going away. I like the feel and tactile response and would lose any natural progression aspect. What I see is that the trackpad itself will not only take complex inputs, but also allow for a visual medium so you can do more things that are easier with direct finger inputs over a more specific medium.

Take a software calculator, for instance. If I'm on my Mac I will grab my iPhone to do a calculation because it's the best option I have since using my finger to to hit buttons is better than using the mouse to click around the screen or using the row of numbers on the keyboard. There are several apps that I use the iPhone for even with a Mac in front of me because it's easier and faster. This just makes sense that Apple would do a parallel move of their touch technology into their Macs. People think that the Mac Tablet would be cool and that futuristic movies have cool gadgets but the reality is that if it's more tedious to use than it's predecessor it won't catch on because that isn't progress, no matter how radical it is.

[SpamSandwich]

[solipsism]

Originally Posted by SpamSandwich

Transparant aluminum (also from that scene) would make Macs more seamless and eco-friendly.

[melgross]

Originally Posted by SpamSandwich

I'm raising my hand. A fully gesture driven computer would be a huge waste of movement and energy. It's inefficient. T

I'm not so sure. This is obviously just the very beginning of this technology. Over time, it could very well evolve into something much better than what we use now.

I do see voice as eventually becoming important, though there would be times and places for its use.

The combination of several concepts used together will make for a very powerful UI.

Remember how people who didn't use a mouse thought how useless it was?

The same thing was thought true for computers.

[melgross]

Originally Posted by solipsism

I'm right there with you. The move to new tech isn't about being cool to look at in a movie, it's about feeling natural. We want to use less effort and get more for the result and moving windows around with wild arm gestures and fancy hand twists isn't going to cut it except in Hollywwood.

You won't be doing that on a small screen. Besides, CNN used a large touch screen during the primaries, and will use it during the elections. Very useful.

What you are describing is a long way off. There has to be baby steps for each transition. As pe a recent RD article, I think that a mutli-touch trackpad (note that they already are multitouch on the MBP and MBA) with a visual output is the next step.

We might see something pretty advanced in ten years. This will take time, but I think it's the future.

[aegisdesign]

Originally Posted by melgross

You won't be doing that on a small screen. Besides, CNN used a large touch screen during the primaries, and will use it during the elections. Very useful.

I've seen those on a few TV news programs. To me it's always looked like the reporter is 'miming' the touch screen and there's actually some tech guy behind the scenes just clicking NEXT on the broadcast equivalent of Powerpoint.

[BB Sting]

Okay, so the screen is gonna be the camera. The computer only boots up if it recognizes your face or fingerprint or voice. Instead of moving your mouse cursor to an icon, you simply look at it, the computer can recognize the difference between a casual glance and an intent look. If it sees an intent look the icon will get bigger. If you wish to open the icon, blink at it. Steve Jobs is more than a man and less than a god.

[dtfitzp]

I predict at multi-touch keyboard that is also mobile tablet/PDA/large iPod touch. Instead of a docking station inside your iMac, when you plunk your tablet down in front of your iMac it syncs and becomes a context specific multi-touch keyboard/giant track pad/input device. When you take it away it is your media player, web-enabled tablet mobile computing solution.

[SpamSandwich]

Originally Posted by melgross

Besides, CNN used a large touch screen during the primaries, and will use it during the elections. Very useful.

I tend to disagree. The CNN device is pure showbiz. There are older technologies that enable an interactive screen (I'm thinking of the old football announcer's screen) that are less theatrical, but just as efficient about getting the point across. And just look at that stupid wipe-off board that Karl Rove has been using....

[melgross]

Originally Posted by aegisdesign

I've seen those on a few TV news programs. To me it's always looked like the reporter is 'miming' the touch screen and there's actually some tech guy behind the scenes just clicking NEXT on the broadcast equivalent of Powerpoint.

I can't speak for other networks, but CNN uses a screen made by one of the pioneers in the business. I think its a company that asian guy set up (forgot his name).

[melgross]

Originally Posted by SpamSandwich

I tend to disagree. The CNN device is pure showbiz. There are older technologies that enable an interactive screen (I'm thinking of the old football announcer's screen) that are less theatrical, but just as efficient about getting the point across. And just look at that stupid wipe-off board that Karl Rove has been using....

I don't know how you can disagree with that. It's a simple fact. The CNN screen works very well and does far more than those old screens did, which was nothing, really.

[solipsism]

Originally Posted by melgross

I don't know how you can disagree with that. It's a simple fact. The CNN screen works very well and does far more than those old screens did, which was nothing, really.

They are cool but I am coming up short trying to find a widespread consumer use for the technology. Only flashy marketing uses come to mind when I think of large, MS Surface-like touchscreens.

Going off topic here...
Even on a smaller scale I find the tablet technology falls apart when it gets too large or becomes the main input for your all your data. Except for health care and few other occupations I don't see how a tablet notebook would make things easier for the majority.

[melgross]

Originally Posted by solipsism

They are cool but I am coming up short trying to find a widespread consumer use for the technology. Only flashy marketing uses come to mind when I think of large, MS Surface-like touchscreens.

Going off topic here...
Even on a smaller scale I find the tablet technology falls apart when it gets too large or becomes the main input for your all your data. Except for health care and few other occupations I don't see how a tablet notebook would make things easier for the majority.

I see no problem, if it's properly done. I'm not wedded to the way we do things now.

I liked my Palm. I liked my old Wacoms, I like my new Wacom Cintiq 12", and I like my new iPhone.

In all these cases, I can do more, easily. I'm really looking forwards to newer methods to use similar devices.

I keep my Cintiq in my lap most of the time, where it works well. I see no problem if that were a multitouch screen, as well as a graphics tablet. I'd love the idea of a key that would switch between graphics and other input usage.

I really believe that some people are having a difficult time getting a grasp of the ease of use, and possibilities.

I can imagine all sorts of ways this could work.

[solipsism]

Originally Posted by melgross

I see no problem, if it's properly done. I'm not wedded to the way we do things now.
[...]
I can imagine all sorts of ways this could work.

I'm not either, I am looking to future and constantly speculating on what I think the next move will be, but it has to feel natural and allow you to be more productive in the end.

— A giant TV with a touch interface that you run your fingers on seems so difficult compared to a mouse, keyboard or remote control.

— A notebook with no physical keyboard just another screen with a virtual keyboard would be murderous on my hands and would require me to look at the keys.

— A notebook with the main screen being a touchscreen would be awkward. I'd either have to swivel it to use it as a touchpad without the use of the keyboard or keep the screen upright which is awkward.

Making the touchpad dynamic with a visual output and more complex inputs that match the output is the only touchscreen transition that seems like a natural evolutionary process. I'd love to hear hear your ideas because I'm coming up short seeing any sort of natural progression with the ideas that are commonly floated around hear.

[melgross]

Originally Posted by solipsism

I'm not either, I am looking to future and constantly speculating on what I think the next move will be, but it has to feel natural and allow you to be more productive in the end.

— A giant TV with a touch interface that you run your fingers on seems so difficult compared to a mouse, keyboard or remote control.

That giant monitor does look to be helpful. I was watching the primaries, and once they got the software fixed, which was glitching in the beginning, the screen was very helpful.

— A notebook with no physical keyboard just another screen with a virtual keyboard would be murderous on my hands and would require me to look at the keys.

Probably, at first, there will be a keyboard, and this will be an addendum to that. A complete new interface is likely years away. In an older thread about this I showed some work that is being done to give a screen keyed surfaces when needed. There is technology being worked on that can, with tiny electrical signals, stretch, and shrink the surface of special materials that could be used for screen surfaces.

By stretching the space around a square area, the surface is intented as an arc. By doing that, there are spaces delineated as keys. With judicious working of the materials, the center of those "keys" can be made to indent when a finger is pressed lightly, by using the capacitive effects of the finger, interacting with the electrically sensitive surface of the key, which is adjusted to respond to it.

Thus, you would have the feel of a keypress, perhaps attended to by a click, or some other suitably annoying attention getting action.

This could even give the bumps found on most keyboards (lacking on Apple's latest, for some reason) for the centering keys, the "F" and the "J".

When the keyboard isn't needed, a press on a button off the screen would make it go away. This could be used for any shape.

There is no reason to believe that anything but good can come from this.

— A notebook with the main screen being a touchscreen would be awkward. I'd either have to swivel it to use it as a touchpad without the use of the keyboard or keep the screen upright which is awkward.

Making the touchpad dynamic with a visual output and more complex inputs that match the output is the only touchscreen transition that seems like a natural evolutionary process. I'd love to hear hear your ideas because I'm coming up short seeing any sort of natural progression with the ideas that are commonly floated around hear.

See above.

[Alan King]

Apple is getting ready for when computers will have entered or system and we will have no keyboards for interacting with them. The computer industry of the future is bound to be Apple dominated.