Apple patents hover-sensing multitouch display

Posted:
in General Discussion edited February 2016
After the release of 3D Touch force-sensing input on iPhone 6s, Apple continues its push for the perfect input technology, most recently winning a patent that uses inline proximity sensors to detect non-contact hover gestures.




Granted by the U.S. Patent and Trademark Office on Tuesday, Apple's U.S. Patent No. 9,250,734 for "Proximity and multi-touch sensor detection and demodulation" details methods by which photodiodes, or other proximity sensing hardware, work in tandem with traditional multitouch displays to essentially shift the user interaction area beyond the screen. In some ways the invention is similar in scope to 3D Touch, but measures input in an opposite direction along the z-axis relative to an iPhone's screen.

Non-contact user interfaces have long been studied as a means of user input, though recent work in the area has concentrated on long-range implementations that rely on cameras and other specialized optical systems. The 3D motion capture technology Apple acquired with its 2013 purchase of PrimeSense, an Israeli firm that worked on first generation Xbox Kinect hardware, is a good example of a modern large scale UI deployment.

With today's patent, Apple proposes a more intimate solution optimized for use in iPhone, MacBook and other portable devices. Like current iOS devices, the invention includes in part a capacitive sensing element disposed throughout an LCD display with multiple piggybacked proximity sensors. This hybrid solution provides a more complete "image" of touch inputs by delivering hover gesture detection.


Source: USPTO


The document suggests use of infrared LEDs and photodiodes, much like the infrared proximity modules iPhone uses for head detection. Light generated by the LED bounces off a user's finger and is captured by the photodiode, which changes current output as a function of received light.

With multiple sensor arrays, installed on a per-pixel basis, in groups or in rows, the system can detect a finger, palm or other object hovering just above a display surface. Further, these photodiodes would be connected to the same analog channels as nearby physical touch sensors, thereby saving space and power.

Translating detected motion to a GUI, users can "push" virtual buttons, trigger functions without touching a display, toggle power to certain hardware components and more. With multiple photodiodes installed, Apple would also be able to do away with the IR proximity sensor currently located next to iPhone's ear speaker.




Apple outlines numerous alternative embodiments in its patent, including proximity sensor placement, ideal component types and sample GUI responses. A separate embodiment describes a MacBook implementation in which ancillary hover-sensing displays augment the typing and trackpad experience.

Enticing as it is, the technology is unlikely to make its way into a shipping device anytime soon. Apple is still looking for unique 3D Touch integrations for its first-party apps, while many third-party developers have yet to take advantage of the pressure-sensing iPhone 6s feature. Introducing yet another input method would muddy the waters.

Apple's hover-sensing display patent was first filed for in March 2015 and credits Steven P. Hotelling and Christoph H. Krah as its inventors.
«1

Comments

  • Reply 1 of 31
    jkichlinejkichline Posts: 1,369member
    The Android trolls are out in full force today, parading around in their little furry Android outfits, trumpeting the victory of the almighty Google and their overlords, Alphabet.  One such participant announced "that it's a fact that Google innovates more then Apple".  I beg to differ. I'd like to see them match this technology.

    Personally, I think Apple is in a lull on purpose as they ramp up the next round of releases this year.  They also don't care if their stock "sinks" a little because we all know they are not going to play Wall Streets little game.  They will wait until the time is right and prove once again that Google is just an advertising company with little side projects to keep smart people interested in them.
    WiseGuybuckaleccornchipsuddenly newtonwonkothesanecali
  • Reply 2 of 31
    mauszmausz Posts: 243member
    jkichline said:
    I beg to differ. I'd like to see them match this technology.

    http://www.samsung.com/uk/support/skp/faq/1053493 

    Different technical implementation and actually removed on the S6 (from TouchWiz)
    edited February 2016 singularity
  • Reply 3 of 31
    mausz said:
    jkichline said:
    I beg to differ. I'd like to see them match this technology.

    http://www.samsung.com/uk/support/skp/faq/1053493 

    Different technical implementation and actually removed on the S6 (from TouchWiz)
    yeh I have a Lenovo tablet that does something similar with "glove mode", I remember an old Sony phone also having it. Samsung Note 10.1 (2013) picks up the stylus hovering from quite a distance, can't remember if it detects finger from same distance however.
  • Reply 4 of 31
    mauszmausz Posts: 243member
    adm1 said:
    mausz said:
    http://www.samsung.com/uk/support/skp/faq/1053493 

    Different technical implementation and actually removed on the S6 (from TouchWiz)
    yeh I have a Lenovo tablet that does something similar with "glove mode", I remember an old Sony phone also having it. Samsung Note 10.1 (2013) picks up the stylus hovering from quite a distance, can't remember if it detects finger from same distance however.
    I believe I read that Samsung was using a 'weakness' of the touch screen. Current starts leaking without actually touching the display. Normally you would have a threshold to determine an actual press. With air view they had 2 thresholds. Works with fingers.

    Sometimes my iPad also detects a touch while I have not touched the display yet (or maybe my finger is not sensitive enough to tell me I actually did ;)
  • Reply 5 of 31
    foggyhillfoggyhill Posts: 4,767member
    mausz said:
    jkichline said:
    I beg to differ. I'd like to see them match this technology.

    http://www.samsung.com/uk/support/skp/faq/1053493 

    Different technical implementation and actually removed on the S6 (from TouchWiz)
    So, another shit implementation from Samsung.. Big surprise... Also, how does that relate in any way to Google?

    cali
  • Reply 6 of 31
    irelandireland Posts: 17,798member
    I think regarding devices the three biggest revolutions in the next 10-20 years will be self-driving cars. Display cameras: where you can look the other person in the eye (among many to-be-discovered benefits). And wireless power that is cost effective, requires no user intervention and works over Wifi distances. Distance-sensing touch targets will be minor considering. Real world benefits of such a feature remain to be seen.
    edited February 2016
  • Reply 7 of 31
    irelandireland Posts: 17,798member
    mausz said:

    Sometimes my iPad also detects a touch while I have not touched the display yet (or maybe my finger is not sensitive enough to tell me I actually did
    Do you play an instrument or have you done much hand physical labour? You may have finger tip numbness.
    edited February 2016
  • Reply 8 of 31
    foggyhill said:
    mausz said:
    http://www.samsung.com/uk/support/skp/faq/1053493 

    Different technical implementation and actually removed on the S6 (from TouchWiz)
    So, another shit implementation from Samsung.. Big surprise... Also, how does that relate in any way to Google?

    Why do you say this? Have you ever tested it?
    Air View was one of the reasons I did not upgrade to Galaxy S6. It just works fine. I have no idea why they have removed it. Unlike Apple Samsung does not create an own TV Ad for each single feature, just because the Galaxys are full of features (not all of them are good) and many users are not aware of them.

    You just should accept that in the meantime Apple has changed from an Innovator to a Follower. Similar to the pop-up windows and multi window features Apple will have again a very good template from Samsung to adopt.

    And since Edge series Samsung has also its own design language, whether or not you like it, 
    stevie
  • Reply 9 of 31
    jaffajaffa Posts: 15member
    jkichline said:
    I'd like to see them match this technology.

    Reflective IR proximity sensors have been used for years (maybe decades) in industry.

    As for mobile phone implementations of similar technology the 2nd (and 3rd) gen MotoX use IR proximity sensors to detect the users hand approaching the screen to wake it up, that predates this patent application by six months.

    Active, reflective IR sensors would be extremely wasteful of energy for a battery operated device.

    What's the innovation here?
    stevie
  • Reply 10 of 31
    mauszmausz Posts: 243member
    ireland said:
    I think regarding devices the three biggest revolutions in the next 10-20 years will be : And wireless power that is cost effective, requires no user intervention and works over Wifi distances.
    Have a look at COTA 

    https://www.youtube.com/watch?v=k8bsg9TpRYk  or in more detail : http://tweakers.net/video/11243/draadloos-laden-via-ossia-cota.html

    Seems interesting. It uses focussed 2.4Ghz and is small enough (as the video shows AA batteries with a receiver and not only the ugly receiver case).

    No information on power efficiency. Power performance can be a problem at max 1W, but if it works as advertised charging a phone in 6~7 hours is no problem
  • Reply 11 of 31
    irelandireland Posts: 17,798member

    mausz said:
    ireland said:
    I think regarding devices the three biggest revolutions in the next 10-20 years will be : And wireless power that is cost effective, requires no user intervention and works over Wifi distances.
    "Have a look at COTA"

    I've seen all of Cota's videos and didn't find them particularly impressive. Up-to 1watt? Sorry but that is no solution. Imagine a day? GTFO. Look at their actual demos, they are far less impressive. I'm taking up to hundreds of watts safely across your whole house. That sort of revolution.
    edited February 2016 cali
  • Reply 12 of 31
    22july201322july2013 Posts: 3,564member
    If hovering can be made to work, does that mean Apple can now support Flash? Here was Steve's exact words:

    http://www.apple.com/hotnews/thoughts-on-flash/
    "Fifth, there’s Touch. Flash was designed for PCs using mice, not for touch screens using fingers. For example, many Flash websites rely on “rollovers”, which pop up menus or other elements when the mouse arrow hovers over a specific spot. Apple’s revolutionary multi-touch interface doesn’t use a mouse, and there is no concept of a rollover."

    Now that there can be " a rollover", I guess this reason for not supporting Flash is no longer valid.

    stevie
  • Reply 13 of 31
    steviestevie Posts: 956member
    jkichline said:
     I'd like to see them match this technology.


    Samsung copied this many years ago.
  • Reply 14 of 31
    mike1mike1 Posts: 3,275member
    ireland said:
    mausz said:

    Sometimes my iPad also detects a touch while I have not touched the display yet (or maybe my finger is not sensitive enough to tell me I actually did
    Do you play an instrument or have you done much hand physical labour? You may have finger tip numbness.
    I too have this happen to me occassionally. I always attribute it to a higher than normal static charge, like if I had touched something else I would have gotten a carpet shock. But who knows.
  • Reply 15 of 31
    williamhwilliamh Posts: 1,032member
    If hovering can be made to work, does that mean Apple can now support Flash? Here was Steve's exact words:

    http://www.apple.com/hotnews/thoughts-on-flash/
    "Fifth, there’s Touch. Flash was designed for PCs using mice, not for touch screens using fingers. For example, many Flash websites rely on “rollovers”, which pop up menus or other elements when the mouse arrow hovers over a specific spot. Apple’s revolutionary multi-touch interface doesn’t use a mouse, and there is no concept of a rollover."

    Now that there can be " a rollover", I guess this reason for not supporting Flash is no longer valid.

    I guess Apple will  rely on the FIVE other great reasons for not supporting Flash cited in the link you thoughtfully provided. Poor reliability, security and performance was ONE reason.
    calinolamacguy
  • Reply 16 of 31
    SoliSoli Posts: 10,035member
    Personally, this is the kind of sensing I've wanted for the dashboard for ages. Static buttons are too limited in capabilities and a touchscreen display has too many unknowns for eyes-free driving. Hover-sensing could help resolve one of the issues with the future of the dashboard.
  • Reply 17 of 31
    cnocbuicnocbui Posts: 3,613member
    stevie said:
    jkichline said:
     I'd like to see them match this technology.


    Samsung copied this many years ago.
    I had my suspicions, well done.
    singularity
  • Reply 18 of 31
    cornchipcornchip Posts: 1,945member
    This would be the final nail in Wacom's coffin
    stevie
  • Reply 19 of 31
    SoliSoli Posts: 10,035member
    cornchip said:
    This would be the final nail in Wacom's coffin
    Could you be more specific? I'd have thought the sophistication of the Pencil would have done that, if anything, but since Apple's tech isn't going to work with Windows I'd think Wacom will still have a significant niche for Windows users.
    stevie
  • Reply 20 of 31
    This sure would've been a nice feature to have on the current iPad Pro.
Sign In or Register to comment.