Apple looks to take multi-touch beyond the touch-screen
With its competitors struggling to catch up with multi-touch technology introduced last year as part of the iPhone, Apple is already conceptualizing new versions of the technology that would fuse a variety of secondary inputs with today's touch-based gestures to produce more efficient data input operations.
A new 30-page patent filing by Wayne Westerman and John Elias, co-founders of the Fingerworks firm acquired by Apple during the development of the original iPhone, details a handful of these newly proposed inputs under the title "Multi-Touch Data Fusion."
The pair of engineers note that while the fingertip chording and movement data generated by today's multi-touch input devices can provide a strong set of user control, fusing additional information from other sensing modalities can significantly enhance the interpretative abilities of a device or significantly improve its overall ease of use.
Among the secondary input means outlined in the filing (and detailed below) are voice fusion, finger identification fusion, gaze vector fusion, biometrics fusion, and facial expression fusion.
Voice Fusion
In the case of Voice Fusion, it's proposed that voice input be applied to actions on a multi-touch device that are poorly served by manual input while manual input handles tasks poorly served by voice.
For example, the filing presents the scenario where modifications to a mechanical drawing require resizing, rotation, and color change. Using voice commands to resize and rotate objects in the drawing would prove problematic because a verbal description of the intended size and rotation would be difficult to express. Therefor, those operations are best suited for manipulation by manual multi-touch gestures.
On the other hand, "using multi-touch to select a color is typically less efficient than using voice because the color has to be selected by traversing a list," Westerman explained. "Alternatively or additionally, voice input may be used to insert text in the object."
Finger Identification Fusion
Today's multi-touch touch sensors are efficient at detecting the presence of human fingers, but they do so with ambiguity as to which specific fingers are making the contacts. While that limitation doesn't pose a problem for existing applications, there are some examples where precise finger identification is imperative.
"Finger painting, where each finger has an assigned color, stroke, or other characteristic, is a simple example of an application that would be significantly enhanced compared to the state-of-the-art by using finger identification with multi-touch data fusion," Westerman wrote. "For example, if the index finger of the left hand is assigned the color red and the other fingers are assigned different colors the application must be able to determine when the index finger of the left hand is in contact with the surface in order to paint red. Conversely, the application must be able to determine when the red-assigned finger is not in contact with the surface. The fusion of finger identification data with multi-touch movement data allows the application to function without error."
To help identify individual fingers, the engineer proposes the user of a digital video camera, like Apple's built-in iSights, to provide a view over the multi-touch surface. The camera data would determine where the fingers of each hand are relative to the multi-touch XY coordinates, while the multi-touch sensor determines when the fingers make contact with the multi-touch surface.
Gaze Vector Fusion
Similarly, iSight cameras could also serve to record gaze vector data, where operations on a computer screen are partially determined by the direction in which a user directs his eyes or head position.
For example, if the user wishes to bring forward a window in the lower left corner of a screen, which is currently underneath two other windows, the user would direct his gaze to the window of interest and then tap a specific chord on the multi-touch surface.
Biometrics Fusion
The filing also suggests the use of biometric input, or input that is determined by hand size, fingerprint input, body temperature, heart rate, skin impedance, and pupil size.
"Typical applications that might benefit from the fusion of biometric data with multi-touch movement data would include games, security, and fitness related activities," Westerman wrote. "Hand characteristics such as size, shape, and general morphology can be used to identify an individual for the purpose of allowing access to secured areas, including computer systems. While hand characteristics alone would not provide a sufficient level of identity verification, it could be the first door through which a user must pass before other security measures are applied."
Facial Expression Fusion
A final means of secondary input outlined in Westerman's filing, which seems years off, would again employ a digital video camera like an iSight, to determine operations based on a user's facial expression and signs of frustration.
For example, he notes that a novice user may experience frustration from time to time when learning how to perform some task with an electronic device. "Say that the user is trying to scroll through a document using a two-finger vertical movement (gesture). Scrolling, however, is not working for him because he is unknowingly touching the surface with three fingers instead of the required two," he wrote. "He becomes frustrated with the 'failure' of the device. However, in this case, the system recognizes the frustration and upon analyzing the multi-touch movement data concludes he is trying to scroll with three fingers. At this point, the device could bring the extra-finger problem to the attention of the user or it could decide to ignore the extra finger and commence scrolling. Subsequent emotional data via facial recognition would confirm to the system that the correct remedial action was taken."
The filing by Westerman and Elias is dated Dec. 27, 2007.
A new 30-page patent filing by Wayne Westerman and John Elias, co-founders of the Fingerworks firm acquired by Apple during the development of the original iPhone, details a handful of these newly proposed inputs under the title "Multi-Touch Data Fusion."
The pair of engineers note that while the fingertip chording and movement data generated by today's multi-touch input devices can provide a strong set of user control, fusing additional information from other sensing modalities can significantly enhance the interpretative abilities of a device or significantly improve its overall ease of use.
Among the secondary input means outlined in the filing (and detailed below) are voice fusion, finger identification fusion, gaze vector fusion, biometrics fusion, and facial expression fusion.
Voice Fusion
In the case of Voice Fusion, it's proposed that voice input be applied to actions on a multi-touch device that are poorly served by manual input while manual input handles tasks poorly served by voice.
For example, the filing presents the scenario where modifications to a mechanical drawing require resizing, rotation, and color change. Using voice commands to resize and rotate objects in the drawing would prove problematic because a verbal description of the intended size and rotation would be difficult to express. Therefor, those operations are best suited for manipulation by manual multi-touch gestures.
On the other hand, "using multi-touch to select a color is typically less efficient than using voice because the color has to be selected by traversing a list," Westerman explained. "Alternatively or additionally, voice input may be used to insert text in the object."
Finger Identification Fusion
Today's multi-touch touch sensors are efficient at detecting the presence of human fingers, but they do so with ambiguity as to which specific fingers are making the contacts. While that limitation doesn't pose a problem for existing applications, there are some examples where precise finger identification is imperative.
"Finger painting, where each finger has an assigned color, stroke, or other characteristic, is a simple example of an application that would be significantly enhanced compared to the state-of-the-art by using finger identification with multi-touch data fusion," Westerman wrote. "For example, if the index finger of the left hand is assigned the color red and the other fingers are assigned different colors the application must be able to determine when the index finger of the left hand is in contact with the surface in order to paint red. Conversely, the application must be able to determine when the red-assigned finger is not in contact with the surface. The fusion of finger identification data with multi-touch movement data allows the application to function without error."
To help identify individual fingers, the engineer proposes the user of a digital video camera, like Apple's built-in iSights, to provide a view over the multi-touch surface. The camera data would determine where the fingers of each hand are relative to the multi-touch XY coordinates, while the multi-touch sensor determines when the fingers make contact with the multi-touch surface.
Gaze Vector Fusion
Similarly, iSight cameras could also serve to record gaze vector data, where operations on a computer screen are partially determined by the direction in which a user directs his eyes or head position.
For example, if the user wishes to bring forward a window in the lower left corner of a screen, which is currently underneath two other windows, the user would direct his gaze to the window of interest and then tap a specific chord on the multi-touch surface.
Biometrics Fusion
The filing also suggests the use of biometric input, or input that is determined by hand size, fingerprint input, body temperature, heart rate, skin impedance, and pupil size.
"Typical applications that might benefit from the fusion of biometric data with multi-touch movement data would include games, security, and fitness related activities," Westerman wrote. "Hand characteristics such as size, shape, and general morphology can be used to identify an individual for the purpose of allowing access to secured areas, including computer systems. While hand characteristics alone would not provide a sufficient level of identity verification, it could be the first door through which a user must pass before other security measures are applied."
Facial Expression Fusion
A final means of secondary input outlined in Westerman's filing, which seems years off, would again employ a digital video camera like an iSight, to determine operations based on a user's facial expression and signs of frustration.
For example, he notes that a novice user may experience frustration from time to time when learning how to perform some task with an electronic device. "Say that the user is trying to scroll through a document using a two-finger vertical movement (gesture). Scrolling, however, is not working for him because he is unknowingly touching the surface with three fingers instead of the required two," he wrote. "He becomes frustrated with the 'failure' of the device. However, in this case, the system recognizes the frustration and upon analyzing the multi-touch movement data concludes he is trying to scroll with three fingers. At this point, the device could bring the extra-finger problem to the attention of the user or it could decide to ignore the extra finger and commence scrolling. Subsequent emotional data via facial recognition would confirm to the system that the correct remedial action was taken."
The filing by Westerman and Elias is dated Dec. 27, 2007.
Comments
Clippy: You look frustrated, can I help you?
User: YArrrgghhhhhh!!!!!
A way to do cut/copy/paste on the iPhone.
The user uses voice commands and clearly prnounces..... "Control - A, Control - VEE"
Oh, so simple.....
There's no efficient equivalent of the "click and hold" feature of a mouse on the iPhone. When copy and paste arrives, I'm betting that it's no picnic.
99.9% of the time, when operating an iPhone, you're holding it in one hand and tapping with the other. Give that other hand a conveniently placed button! Apple's already realized that the iTouch needs external buttons --it's not always convenient to have EVERYTHING handled by onscreen touch control. I don't think the iPod experience on the iPhone is as good as a click wheel endowed iPod.
gc
Finally!
A way to do cut/copy/paste on the iPhone.
The user uses voice commands and clearly prnounces..... "Control - A, Control - VEE"
Oh, so simple.....
That would be horrible. Think of some of the horrible key combinations. I'm hoping that copy and paste commands are done very similar to how the maker of MagicPad envisioned it.
If you thought that what you saw was not awesome, raise your hand. Not a single hand goes up!
And what if you want your son to do something on the computer for you? "My dad says it's okay! I even have his authorization code!" iMac: "I'm sorry, I can't let you do that Johnny. Your hands are too little."
All I want is to tell my iMac "define disestablishment" and have Dictionary pop up with the definition.
As it stands, speech recognition sucks. I tell the computer "Open Pages" and it says "Opening iTunes" lol. And it hardly ever responds when I tell it to move a piece in Chess.
I'm dreaming of the Star Trek computer that you can just speak naturally to, and that doesn't have to be "trained" for a specific person. But I don't think voice recognition is anywhere near that level yet.
Actually, a lot of very good stuff doesn't need training and that is getting better all the time.
Just try to call Vonage for support and you get to use an untrained voice recognition system that does very good. (but is still as annoying as all menu driven call center systems)
Check out the move Bladerunner.
Your computer's going to watch your eye and decide if you're an organism that can be trusted by the collective. (sorry too many references there)
All I want is to tell my iMac "define disestablishment" and have Dictionary pop up with the definition.
You can't do that but there is an easier way to get access to the Dictionary for very quick referencing. The current use of the mini-dictionary within most apps requires a complex use of keys but if you map it to an F-key it makes it a toggle key so you don't have to hold that combo down and you can now move around your desktop freely with the mouse and any word it crosses over will pop up instantly with a definition.
"Dave, although you took thorough precautions in the pod against my hearing you, I could see your lips move." "
Very nice....
You can't do that but there is an easier way to get access to the Dictionary for very quick referencing. The current use of the mini-dictionary within most apps requires a complex use of keys but if you map it to an F-key it makes it a toggle key so you don't have to hold that combo down and you can now move around your desktop freely with the mouse and any word it crosses over will pop up instantly with a definition.
I think that should be how it is by default tho, with an easier way to change it.
I think that should be how it is by default tho, with an easier way to change it.
Me too. It's one of the first thing I change on every Mac I set up. I think they need to include one of those nice little printed icons on the PF key to denote what it is, too. It's just so dang useful!
Just watch Tom Cruise in "Minority Report" and you get a sense of where this is going. Seriously.
If you thought that what you saw was not awesome, raise your hand. Not a single hand goes up!
I'm raising my hand. A fully gesture driven computer would be a huge waste of movement and energy. It's inefficient. The Minority Report interface looked the way it did because of Jeff Hahn's work and because Tom Cruise was supposed to look like a conductor in complete command of his craft.
I'm raising my hand. A fully gesture driven computer would be a huge waste of movement and energy. It's inefficient. The Minority Report interface looked the way it did because of Jeff Hahn's work and because Tom Cruise was supposed to look like a conductor in complete command of his craft.
I'm right there with you. The move to new tech isn't about being cool to look at in a movie, it's about feeling natural. We want to use less effort and get more for the result and moving windows around with wild arm gestures and fancy hand twists isn't going to cut it except in Hollywwood.
Actually this IS exciting. I think this is where the future of user interfaces lies. Computers that control our every move, our minute emotions and sense our heartbeat and the slightest tremors. Yep, that's the future and I'm glad it's finally here.
What you are describing is a long way off. There has to be baby steps for each transition. As pe a recent RD article, I think that a mutli-touch trackpad (note that they already are multitouch on the MBP and MBA) with a visual output is the next step.