Apple looks to take multi-touch beyond the touch-screen

Posted:
in Future Apple Hardware edited January 2014
With its competitors struggling to catch up with multi-touch technology introduced last year as part of the iPhone, Apple is already conceptualizing new versions of the technology that would fuse a variety of secondary inputs with today's touch-based gestures to produce more efficient data input operations.



A new 30-page patent filing by Wayne Westerman and John Elias, co-founders of the Fingerworks firm acquired by Apple during the development of the original iPhone, details a handful of these newly proposed inputs under the title "Multi-Touch Data Fusion."



The pair of engineers note that while the fingertip chording and movement data generated by today's multi-touch input devices can provide a strong set of user control, fusing additional information from other sensing modalities can significantly enhance the interpretative abilities of a device or significantly improve its overall ease of use.



Among the secondary input means outlined in the filing (and detailed below) are voice fusion, finger identification fusion, gaze vector fusion, biometrics fusion, and facial expression fusion.



Voice Fusion



In the case of Voice Fusion, it's proposed that voice input be applied to actions on a multi-touch device that are poorly served by manual input while manual input handles tasks poorly served by voice.



For example, the filing presents the scenario where modifications to a mechanical drawing require resizing, rotation, and color change. Using voice commands to resize and rotate objects in the drawing would prove problematic because a verbal description of the intended size and rotation would be difficult to express. Therefor, those operations are best suited for manipulation by manual multi-touch gestures.







On the other hand, "using multi-touch to select a color is typically less efficient than using voice because the color has to be selected by traversing a list," Westerman explained. "Alternatively or additionally, voice input may be used to insert text in the object."



Finger Identification Fusion



Today's multi-touch touch sensors are efficient at detecting the presence of human fingers, but they do so with ambiguity as to which specific fingers are making the contacts. While that limitation doesn't pose a problem for existing applications, there are some examples where precise finger identification is imperative.



"Finger painting, where each finger has an assigned color, stroke, or other characteristic, is a simple example of an application that would be significantly enhanced compared to the state-of-the-art by using finger identification with multi-touch data fusion," Westerman wrote. "For example, if the index finger of the left hand is assigned the color red and the other fingers are assigned different colors the application must be able to determine when the index finger of the left hand is in contact with the surface in order to paint red. Conversely, the application must be able to determine when the red-assigned finger is not in contact with the surface. The fusion of finger identification data with multi-touch movement data allows the application to function without error."







To help identify individual fingers, the engineer proposes the user of a digital video camera, like Apple's built-in iSights, to provide a view over the multi-touch surface. The camera data would determine where the fingers of each hand are relative to the multi-touch XY coordinates, while the multi-touch sensor determines when the fingers make contact with the multi-touch surface.



Gaze Vector Fusion



Similarly, iSight cameras could also serve to record gaze vector data, where operations on a computer screen are partially determined by the direction in which a user directs his eyes or head position.







For example, if the user wishes to bring forward a window in the lower left corner of a screen, which is currently underneath two other windows, the user would direct his gaze to the window of interest and then tap a specific chord on the multi-touch surface.



Biometrics Fusion



The filing also suggests the use of biometric input, or input that is determined by hand size, fingerprint input, body temperature, heart rate, skin impedance, and pupil size.



"Typical applications that might benefit from the fusion of biometric data with multi-touch movement data would include games, security, and fitness related activities," Westerman wrote. "Hand characteristics such as size, shape, and general morphology can be used to identify an individual for the purpose of allowing access to secured areas, including computer systems. While hand characteristics alone would not provide a sufficient level of identity verification, it could be the first door through which a user must pass before other security measures are applied."



Facial Expression Fusion



A final means of secondary input outlined in Westerman's filing, which seems years off, would again employ a digital video camera like an iSight, to determine operations based on a user's facial expression and signs of frustration.







For example, he notes that a novice user may experience frustration from time to time when learning how to perform some task with an electronic device. "Say that the user is trying to scroll through a document using a two-finger vertical movement (gesture). Scrolling, however, is not working for him because he is unknowingly touching the surface with three fingers instead of the required two," he wrote. "He becomes frustrated with the 'failure' of the device. However, in this case, the system recognizes the frustration and upon analyzing the multi-touch movement data concludes he is trying to scroll with three fingers. At this point, the device could bring the extra-finger problem to the attention of the user or it could decide to ignore the extra finger and commence scrolling. Subsequent emotional data via facial recognition would confirm to the system that the correct remedial action was taken."



The filing by Westerman and Elias is dated Dec. 27, 2007.
«1

Comments

  • Reply 1 of 37
    I can see it now, in Office 2020:



    Clippy: You look frustrated, can I help you?

    User: YArrrgghhhhhh!!!!!
  • Reply 2 of 37
    eaieai Posts: 417member
    What about hearing the user shout at the computer? Detecting tone of voice etc
  • Reply 3 of 37
    Finally!

    A way to do cut/copy/paste on the iPhone.

    The user uses voice commands and clearly prnounces..... "Control - A, Control - VEE"



    Oh, so simple.....
  • Reply 4 of 37
    I know we all want to be touchy-feely with the iPhone, but how about a simple external modifier key? Put it on the opposite side of the iPhone from the volume buttons. Heck make it two keys like Shift and Command!



    There's no efficient equivalent of the "click and hold" feature of a mouse on the iPhone. When copy and paste arrives, I'm betting that it's no picnic.



    99.9% of the time, when operating an iPhone, you're holding it in one hand and tapping with the other. Give that other hand a conveniently placed button! Apple's already realized that the iTouch needs external buttons --it's not always convenient to have EVERYTHING handled by onscreen touch control. I don't think the iPod experience on the iPhone is as good as a click wheel endowed iPod.



    gc
  • Reply 5 of 37
    All very interesting, but hopefully patents applied for that they never intend to use in real products. The only recent idea I've been taken with is the idea of a multi-touch mouse or a keyboard that adds some kind of mulit-touch and yet retains the feel of a keyboard. At least then the fingers can remain in a good position to continue to do work. Any time the hands even leave the keyboard to go to the mouse is wasted movement.
  • Reply 6 of 37
    solipsismsolipsism Posts: 25,726member
    Quote:
    Originally Posted by wbrasington View Post


    Finally!

    A way to do cut/copy/paste on the iPhone.

    The user uses voice commands and clearly prnounces..... "Control - A, Control - VEE"



    Oh, so simple.....



    That would be horrible. Think of some of the horrible key combinations. I'm hoping that copy and paste commands are done very similar to how the maker of MagicPad envisioned it.
  • Reply 7 of 37
    Just watch Tom Cruise in "Minority Report" and you get a sense of where this is going. Seriously.



    If you thought that what you saw was not awesome, raise your hand. Not a single hand goes up!
  • Reply 8 of 37
    lafelafe Posts: 252member
    Smile, and your iMac smiles with you.
  • Reply 9 of 37
    Some of that is kind of creepy, a computer that senses so much about you...



    And what if you want your son to do something on the computer for you? "My dad says it's okay! I even have his authorization code!" iMac: "I'm sorry, I can't let you do that Johnny. Your hands are too little."



    All I want is to tell my iMac "define disestablishment" and have Dictionary pop up with the definition.



    As it stands, speech recognition sucks. I tell the computer "Open Pages" and it says "Opening iTunes" lol. And it hardly ever responds when I tell it to move a piece in Chess.
  • Reply 10 of 37
    I'm dreaming of the Star Trek computer that you can just speak naturally to, and that doesn't have to be "trained" for a specific person. But I don't think voice recognition is anywhere near that level yet.
  • Reply 11 of 37
    Quote:
    Originally Posted by FuturePastNow View Post


    I'm dreaming of the Star Trek computer that you can just speak naturally to, and that doesn't have to be "trained" for a specific person. But I don't think voice recognition is anywhere near that level yet.



    Actually, a lot of very good stuff doesn't need training and that is getting better all the time.

    Just try to call Vonage for support and you get to use an untrained voice recognition system that does very good. (but is still as annoying as all menu driven call center systems)



    Check out the move Bladerunner.

    Your computer's going to watch your eye and decide if you're an organism that can be trusted by the collective. (sorry too many references there)
  • Reply 12 of 37
    "Dave, although you took thorough precautions in the pod against my hearing you, I could see your lips move." "
  • Reply 13 of 37
    solipsismsolipsism Posts: 25,726member
    Quote:
    Originally Posted by macosxp View Post


    All I want is to tell my iMac "define disestablishment" and have Dictionary pop up with the definition.



    You can't do that but there is an easier way to get access to the Dictionary for very quick referencing. The current use of the mini-dictionary within most apps requires a complex use of keys but if you map it to an F-key it makes it a toggle key so you don't have to hold that combo down and you can now move around your desktop freely with the mouse and any word it crosses over will pop up instantly with a definition.



  • Reply 14 of 37
    Quote:
    Originally Posted by 8CoreWhore View Post


    "Dave, although you took thorough precautions in the pod against my hearing you, I could see your lips move." "



    Very nice....
  • Reply 15 of 37
    Quote:
    Originally Posted by solipsism View Post


    You can't do that but there is an easier way to get access to the Dictionary for very quick referencing. The current use of the mini-dictionary within most apps requires a complex use of keys but if you map it to an F-key it makes it a toggle key so you don't have to hold that combo down and you can now move around your desktop freely with the mouse and any word it crosses over will pop up instantly with a definition.



    I think that should be how it is by default tho, with an easier way to change it.
  • Reply 16 of 37
    ivladivlad Posts: 742member
    I just love how much Apple in inventing and patent everything. They will always be leader as long as they have the best people working for them.
  • Reply 17 of 37
    solipsismsolipsism Posts: 25,726member
    Quote:
    Originally Posted by trevorlsciact View Post


    I think that should be how it is by default tho, with an easier way to change it.



    Me too. It's one of the first thing I change on every Mac I set up. I think they need to include one of those nice little printed icons on the PF key to denote what it is, too. It's just so dang useful!
  • Reply 18 of 37
    Quote:
    Originally Posted by macinthe408 View Post


    Just watch Tom Cruise in "Minority Report" and you get a sense of where this is going. Seriously.



    If you thought that what you saw was not awesome, raise your hand. Not a single hand goes up!



    I'm raising my hand. A fully gesture driven computer would be a huge waste of movement and energy. It's inefficient. The Minority Report interface looked the way it did because of Jeff Hahn's work and because Tom Cruise was supposed to look like a conductor in complete command of his craft.
  • Reply 19 of 37
    buckbuck Posts: 293member
    Actually this IS exciting. I think this is where the future of user interfaces lies. Computers that control our every move, our minute emotions and sense our heartbeat and the slightest tremors. Yep, that's the future and I'm glad it's finally here.
  • Reply 20 of 37
    solipsismsolipsism Posts: 25,726member
    Quote:
    Originally Posted by SpamSandwich View Post


    I'm raising my hand. A fully gesture driven computer would be a huge waste of movement and energy. It's inefficient. The Minority Report interface looked the way it did because of Jeff Hahn's work and because Tom Cruise was supposed to look like a conductor in complete command of his craft.



    I'm right there with you. The move to new tech isn't about being cool to look at in a movie, it's about feeling natural. We want to use less effort and get more for the result and moving windows around with wild arm gestures and fancy hand twists isn't going to cut it except in Hollywwood.



    Quote:
    Originally Posted by Buck View Post


    Actually this IS exciting. I think this is where the future of user interfaces lies. Computers that control our every move, our minute emotions and sense our heartbeat and the slightest tremors. Yep, that's the future and I'm glad it's finally here.



    What you are describing is a long way off. There has to be baby steps for each transition. As pe a recent RD article, I think that a mutli-touch trackpad (note that they already are multitouch on the MBP and MBA) with a visual output is the next step.
Sign In or Register to comment.