Apple's virtual input tool patent could bring next-level UI interactivity to OS X

Posted:
in General Discussion edited January 2014
An Apple patent granted by the U.S. Patent and Trademark Office on Tuesday describes a system in which a "virtual input," such as a trackpad, is recreated onscreen with various interactive objects, allowing users to quickly navigate programs and documents.

Virtual Input
Source: USPTO


Apple's U.S. Patent No. 8,427,438 for "Virtual input tools" is a simple, yet effective tweak to the current user interfaces offered in modern computer operating systems. Unlike a traditional UI, which translates data from an input device and maps it to the movements of a cursor, the patent outlines a system that allows for more direct interactivity.

The patent first calls for a virtual representation of a physical input device, such as the multitouch trackpad found in Apple's MacBook Pro, to be displayed onscreen. Mapped to the coordinates of the virtual trackpad are the corresponding coordinates of the actual device, which in some embodiments are in proportion to the digital version. Presented within the virtual trackpad's boundaries are various interactive elements that can be directly selected and manipulated without searching for a cursor as they are represented in direct relation to the physical device. According to the patent, interactive objects can be application windows, directories and open media files, among others.

From the patent summary:
A virtual representation of an input device can be a two-dimensional area that increases an amount of data (e.g., virtual representations of objects) that can be presented at a particular time, thereby improving the user's experience.
By implementing this method, the entirety of a trackpad's surface can be effectively utilized to navigate a corresponding virtual environment. As noted in the patent language, a user can also use multitouch gestures, like one-finger tap-and-hold, to further enhance the system.

Virtual Input
Illustration of virtual trackpad.


Six engines drive the virtual system: an identification engine that detects input on the physical device; a render engine that draws the virtual interface tools and/or content; a mapping engine to correlate physical coordinates with virtual coordinates; a preference engine to customize the virtual device; an interactivity engine that processes user interactions and dictates how they are translated onscreen; and a presentation engine that creates interactive object representations on the virtual device.

Perhaps most important of the six, and the most helpful in explaining the patent, is the interactivity engine. In some embodiments, the engine will receive a signal that a user has touched or tapped a certain area of the trackpad. From this information, the area which was touched is mapped to the virtual trackpad, and the interactive object associated with that space is activated.

Virtual Input
Selecting the top left object highlights the asset.



In another example, a multitouch gesture such as a swipe is recognized. The associated object on the virtual device is manipulated according to a predefined set of rules, which could be the rendering of an interface window in the quadrant selected. Multitouch gestures can also be used to control the size and operation of the virtual display. For example, a four-finger swipe can minimize the virtual view, while another input can enlarge the interface.

Finally, a control object can be utilized and integrated into the virtual trackpad to deactivate or exit out of the UI. This mechanism can also be used in conjunction with other areas of the virtual tool. For example, a user can manipulate interactive content without making permanent changes if their finger is in continuous contact with the control object. Once a user's finger is lifted off this area, interaction is canceled.

Virtual Input
Control object (318) highlighted.


As with many Apple patents, it is unclear if the virtual input device will make its way to a consumer product, but with a push toward multitouch input, the technology could be a useful stopgap if the company decides to create a laptop with a touchscreen display.

Apple's virtual tool patent was first filed for in 2009 and credits John O. Louch as its inventor.

Comments

  • Reply 1 of 20
    kovacmkovacm Posts: 57member
    heh... Daniel Eran Dilger (from 2008) "A Product Transition: Giving MacBooks the iPhone Touch"

    http://www.roughlydrafted.com/2008/09/01/a-product-transition-giving-macbooks-the-iphone-touch/

  • Reply 2 of 20
    just make the trackpad a touch screen and call it a day.
  • Reply 3 of 20
    danielswdanielsw Posts: 905member
    kovacm wrote: »
    heh... Daniel Eran Dilger (from 2008) "A Product Transition: Giving MacBooks the iPhone Touch"

    http://www.roughlydrafted.com/2008/09/01/a-product-transition-giving-macbooks-the-iphone-touch/

    Today's new patent would seem to be a significant refinement of that "LCD Trackpad" concept--a much better idea, because it eliminates the hardware cost of the second LCD, as well as not obsoleting so much existing hardware. It could also be used with desktop Mac via the Magic Trackpad. This virtual trackpad would be easier to see, and would be adjustable accessibility-wise and localizing-wise.
  • Reply 4 of 20
    danielswdanielsw Posts: 905member
    just make the trackpad a touch screen and call it a day.

    No. You don't get it. This virtual trackpad would still be controlled by the physical trackpad. This doesn't change the fact that touchscreens on laptops and desktops are awkward for the most part and would incur significant hardware costs.
  • Reply 5 of 20
    tallest skiltallest skil Posts: 43,399member


    Originally Posted by OllieWallieWhiskers View Post

    just make the trackpad a touch screen and call it a day.


     


    Worst. Idea. Ever.

  • Reply 6 of 20
    Looks/sounds cool. As an aside, there seems to be enough visual detail there for competitors to swipe.....heh heh...
  • Reply 7 of 20
    This is simply a potential step in merging future iPads with MBA and merging iOS with OS X. Living in a post-PC world, the need for both power of a PC and touch of the tablet is critical.
  • Reply 8 of 20
    kerrybkerryb Posts: 270member
    Am I the only one here that doesn't get it? Or the only one to admit it?
  • Reply 9 of 20
    razorpitrazorpit Posts: 918member

    Quote:

    Originally Posted by OllieWallieWhiskers View Post



    just make the trackpad a touch screen and call it a day.


    Bad idea.  While the "podcasting elite" all want a touch interface, touch is not something you would not want to use in a business setting.  Touch works great for those 30 - 40 minute intervals but won't work for a 8 - 10 hour day.  Several people around (above) me have tried to make it through a work day with an iPad and keyboard and while technically it can do all the functions they need, they have all said they'd rather have a MacBook Air at the end of the day.


     


    I see this new interface as a way to phase out all Core2 Duo's in 10.9(?).

  • Reply 10 of 20
    anonymouseanonymouse Posts: 6,570member

    Quote:

    Originally Posted by waldobushman View Post



    This is simply a potential step in merging future iPads with MBA and merging iOS with OS X. Living in a post-PC world, the need for both power of a PC and touch of the tablet is critical.


     


    Wrong. You clearly didn't understand what you read. This has absolutely nothing to do with that, and there is a) zero evidence to believe that will happen and b) zero reason to believe that will happen. What this is is another step in the evolution of the functionality of separate input devices and desktop UI. If anything, it points directly in the opposite direction of Apple looking to merge the iOS and OS X UIs and devices, and toward a commitment to each continuing develop along it's own evolutionary path. Separate species with a common ancestor.

  • Reply 11 of 20
    A revolutionary pc is coming .
  • Reply 12 of 20
    It seems to me that everyone is missing a VERY likely use for this patent... The new AppleTV Set (when it comes out). This is the TV UI puzzle Steve said he solved!

    Think about it... Changing channels and selecting items on TV, even navigating menus on the current AppleTV requires SCROLLING SLOWLY, PAINFULLY, through linear menus. This would map your iPad, iPhone, or similar "remote" shipped with the TV to the screen so you could DIRECTLY SELECT among a large list or set of movies, shows, channels... Whatever! FAST!

    What do you all think?
  • Reply 13 of 20

    Quote:

    Originally Posted by DigiThinkIT View Post



    It seems to me that everyone is missing a VERY likely use for this patent... The new AppleTV Set (when it comes out). This is the TV UI puzzle Steve said he solved!


     


    Quote:

    Originally Posted by crazy_mac_lover View Post



    A revolutionary pc is coming .


    I think both are spot on.

  • Reply 14 of 20
    Quote:
    Originally Posted by DanielSW View Post



    Today's new patent would seem to be a significant refinement of that "LCD Trackpad" concept--a much better idea, because it eliminates the hardware cost of the second LCD, as well as not obsoleting so much existing hardware. It could also be used with desktop Mac via the Magic Trackpad. This virtual trackpad would be easier to see, and would be adjustable accessibility-wise and localizing-wise.

     

    I was visualizing working with this in my head (I have a knack for that) and I could see that it was amazingly useful to improve interaction.

     

    As a trained artist (among other things) I'm used to having the ability to work with my hands without looking at them -- though it's fairly tough for most people to do. Hence the practice people need with touch typing and not everyone gets it. Having a HUD (Heads Up Display) for the trackpad virtually on the screen would improve people's ability to interact and NOT look at their hands -- and have a better idea spatially of the results of that interaction.

     

    I predict that this will become a "standard" for kiosks and other interaction situations where the User is likely to NOT have a huge amount of experience with the device. Most people take time to get acquainted with keyboards and trackpads with only a slight difference in orientation. Subtle feedback on devices always enhances their accessibility -- and this is one of those GOOD ideas to improve computer interface.
  • Reply 15 of 20
    jasenj1jasenj1 Posts: 912member


    I think this is a good idea and I'm surprised there isn't lots of prior art.


     


    Making the touchpad an LCD is bad because 1) your hands cover it. 2) your focus is off the screen and onto the touchpad.


    Putting a virtual touchpad on the screen works because our brains can easily do the mapping between the physical device and the screen - like we do with a mouse.


     


    The difficulty with the proposed approach is how much of the screen is covered by the virtual touchpad? You'd have to be very careful to make sure the working area of the virtual touchpad provides valuable controls while not obscuring stuff the user needs to see - think documents in multiple windows and reading stuff in one window needed in another window (instructions, reference material, etc.) Also you'll have to make the target regions the "right" size. Looking at the touchpad on my machine, I'd think 5 regions wide by 4 high might be about right. And switching from mouse/pointer to virtual touchpad would have to be done carefully - what if you have the virtual touchpad open but want to scroll or move a window on the desktop?


     


    Imagine an application like ProTools where an equalizer plug-in could pop-up and map its controls to the touchpad - directly change the EQ curve, or raise and lower sliders using the virtual touchpad. Or in Photoshop a color blender control could pop-up.


     


    I think linear motions (vs turning virtual knobs) and heavy use of the edges and corners would make sense.


     


    This could be a fun technology to have integrated at the OS level - available to apps to use and abuse as they will.


     


    - Jasen.

  • Reply 16 of 20
    barleybarley Posts: 10member
    If you focus a camera from the open screen on the desk where the users hand is, it could be a virtual mouse, just by moving the hand across the table and tapping it with fingers, instead of a real mouse.
  • Reply 17 of 20
    evilutionevilution Posts: 1,347member


    I see a lot of people saying NO but I really don't see the issue of having a small screen underneath a glass track pad.

  • Reply 18 of 20
    anonymouseanonymouse Posts: 6,570member

    Quote:

    Originally Posted by Evilution View Post


    I see a lot of people saying NO but I really don't see the issue of having a small screen underneath a glass track pad.



     


    You need to have good haptic feedback to make this truly useful. Eventually, something like this will replace the keyboard and the trackpad.

  • Reply 19 of 20
    tallest skiltallest skil Posts: 43,399member


    Originally Posted by Evilution View Post

    I see a lot of people saying NO but I really don't see the issue of having a small screen underneath a glass track pad.


     


    Boils down to one word: point.

  • Reply 20 of 20
    Doesn't windows have this of a keyboard?
    This is simply a potential step in merging future iPads with MBA and merging iOS with OS X. Living in a post-PC world, the need for both power of a PC and touch of the tablet is critical.
    A Mac tablet is what I would like. They could have it with OS X and IOS with the power for both reaching at a screen size of about 12 inches.(could run windows 8. /s) how to switch well tablet boot camp.
Sign In or Register to comment.