Apple's virtual input tool patent could bring next-level UI interactivity to OS X
An Apple patent granted by the U.S. Patent and Trademark Office on Tuesday describes a system in which a "virtual input," such as a trackpad, is recreated onscreen with various interactive objects, allowing users to quickly navigate programs and documents.
Source: USPTO
Apple's U.S. Patent No. 8,427,438 for "Virtual input tools" is a simple, yet effective tweak to the current user interfaces offered in modern computer operating systems. Unlike a traditional UI, which translates data from an input device and maps it to the movements of a cursor, the patent outlines a system that allows for more direct interactivity.
The patent first calls for a virtual representation of a physical input device, such as the multitouch trackpad found in Apple's MacBook Pro, to be displayed onscreen. Mapped to the coordinates of the virtual trackpad are the corresponding coordinates of the actual device, which in some embodiments are in proportion to the digital version. Presented within the virtual trackpad's boundaries are various interactive elements that can be directly selected and manipulated without searching for a cursor as they are represented in direct relation to the physical device. According to the patent, interactive objects can be application windows, directories and open media files, among others.
From the patent summary:
Illustration of virtual trackpad.
Six engines drive the virtual system: an identification engine that detects input on the physical device; a render engine that draws the virtual interface tools and/or content; a mapping engine to correlate physical coordinates with virtual coordinates; a preference engine to customize the virtual device; an interactivity engine that processes user interactions and dictates how they are translated onscreen; and a presentation engine that creates interactive object representations on the virtual device.
Perhaps most important of the six, and the most helpful in explaining the patent, is the interactivity engine. In some embodiments, the engine will receive a signal that a user has touched or tapped a certain area of the trackpad. From this information, the area which was touched is mapped to the virtual trackpad, and the interactive object associated with that space is activated.
Selecting the top left object highlights the asset.
In another example, a multitouch gesture such as a swipe is recognized. The associated object on the virtual device is manipulated according to a predefined set of rules, which could be the rendering of an interface window in the quadrant selected. Multitouch gestures can also be used to control the size and operation of the virtual display. For example, a four-finger swipe can minimize the virtual view, while another input can enlarge the interface.
Finally, a control object can be utilized and integrated into the virtual trackpad to deactivate or exit out of the UI. This mechanism can also be used in conjunction with other areas of the virtual tool. For example, a user can manipulate interactive content without making permanent changes if their finger is in continuous contact with the control object. Once a user's finger is lifted off this area, interaction is canceled.
Control object (318) highlighted.
As with many Apple patents, it is unclear if the virtual input device will make its way to a consumer product, but with a push toward multitouch input, the technology could be a useful stopgap if the company decides to create a laptop with a touchscreen display.
Apple's virtual tool patent was first filed for in 2009 and credits John O. Louch as its inventor.
Source: USPTO
Apple's U.S. Patent No. 8,427,438 for "Virtual input tools" is a simple, yet effective tweak to the current user interfaces offered in modern computer operating systems. Unlike a traditional UI, which translates data from an input device and maps it to the movements of a cursor, the patent outlines a system that allows for more direct interactivity.
The patent first calls for a virtual representation of a physical input device, such as the multitouch trackpad found in Apple's MacBook Pro, to be displayed onscreen. Mapped to the coordinates of the virtual trackpad are the corresponding coordinates of the actual device, which in some embodiments are in proportion to the digital version. Presented within the virtual trackpad's boundaries are various interactive elements that can be directly selected and manipulated without searching for a cursor as they are represented in direct relation to the physical device. According to the patent, interactive objects can be application windows, directories and open media files, among others.
From the patent summary:
By implementing this method, the entirety of a trackpad's surface can be effectively utilized to navigate a corresponding virtual environment. As noted in the patent language, a user can also use multitouch gestures, like one-finger tap-and-hold, to further enhance the system.A virtual representation of an input device can be a two-dimensional area that increases an amount of data (e.g., virtual representations of objects) that can be presented at a particular time, thereby improving the user's experience.
Illustration of virtual trackpad.
Six engines drive the virtual system: an identification engine that detects input on the physical device; a render engine that draws the virtual interface tools and/or content; a mapping engine to correlate physical coordinates with virtual coordinates; a preference engine to customize the virtual device; an interactivity engine that processes user interactions and dictates how they are translated onscreen; and a presentation engine that creates interactive object representations on the virtual device.
Perhaps most important of the six, and the most helpful in explaining the patent, is the interactivity engine. In some embodiments, the engine will receive a signal that a user has touched or tapped a certain area of the trackpad. From this information, the area which was touched is mapped to the virtual trackpad, and the interactive object associated with that space is activated.
Selecting the top left object highlights the asset.
In another example, a multitouch gesture such as a swipe is recognized. The associated object on the virtual device is manipulated according to a predefined set of rules, which could be the rendering of an interface window in the quadrant selected. Multitouch gestures can also be used to control the size and operation of the virtual display. For example, a four-finger swipe can minimize the virtual view, while another input can enlarge the interface.
Finally, a control object can be utilized and integrated into the virtual trackpad to deactivate or exit out of the UI. This mechanism can also be used in conjunction with other areas of the virtual tool. For example, a user can manipulate interactive content without making permanent changes if their finger is in continuous contact with the control object. Once a user's finger is lifted off this area, interaction is canceled.
Control object (318) highlighted.
As with many Apple patents, it is unclear if the virtual input device will make its way to a consumer product, but with a push toward multitouch input, the technology could be a useful stopgap if the company decides to create a laptop with a touchscreen display.
Apple's virtual tool patent was first filed for in 2009 and credits John O. Louch as its inventor.
Comments
http://www.roughlydrafted.com/2008/09/01/a-product-transition-giving-macbooks-the-iphone-touch/
Today's new patent would seem to be a significant refinement of that "LCD Trackpad" concept--a much better idea, because it eliminates the hardware cost of the second LCD, as well as not obsoleting so much existing hardware. It could also be used with desktop Mac via the Magic Trackpad. This virtual trackpad would be easier to see, and would be adjustable accessibility-wise and localizing-wise.
No. You don't get it. This virtual trackpad would still be controlled by the physical trackpad. This doesn't change the fact that touchscreens on laptops and desktops are awkward for the most part and would incur significant hardware costs.
Originally Posted by OllieWallieWhiskers
just make the trackpad a touch screen and call it a day.
Worst. Idea. Ever.
Quote:
Originally Posted by OllieWallieWhiskers
just make the trackpad a touch screen and call it a day.
Bad idea. While the "podcasting elite" all want a touch interface, touch is not something you would not want to use in a business setting. Touch works great for those 30 - 40 minute intervals but won't work for a 8 - 10 hour day. Several people around (above) me have tried to make it through a work day with an iPad and keyboard and while technically it can do all the functions they need, they have all said they'd rather have a MacBook Air at the end of the day.
I see this new interface as a way to phase out all Core2 Duo's in 10.9(?).
Quote:
Originally Posted by waldobushman
This is simply a potential step in merging future iPads with MBA and merging iOS with OS X. Living in a post-PC world, the need for both power of a PC and touch of the tablet is critical.
Wrong. You clearly didn't understand what you read. This has absolutely nothing to do with that, and there is a) zero evidence to believe that will happen and b) zero reason to believe that will happen. What this is is another step in the evolution of the functionality of separate input devices and desktop UI. If anything, it points directly in the opposite direction of Apple looking to merge the iOS and OS X UIs and devices, and toward a commitment to each continuing develop along it's own evolutionary path. Separate species with a common ancestor.
Think about it... Changing channels and selecting items on TV, even navigating menus on the current AppleTV requires SCROLLING SLOWLY, PAINFULLY, through linear menus. This would map your iPad, iPhone, or similar "remote" shipped with the TV to the screen so you could DIRECTLY SELECT among a large list or set of movies, shows, channels... Whatever! FAST!
What do you all think?
Quote:
Originally Posted by DigiThinkIT
It seems to me that everyone is missing a VERY likely use for this patent... The new AppleTV Set (when it comes out). This is the TV UI puzzle Steve said he solved!
Quote:
Originally Posted by crazy_mac_lover
A revolutionary pc is coming .
I think both are spot on.
Today's new patent would seem to be a significant refinement of that "LCD Trackpad" concept--a much better idea, because it eliminates the hardware cost of the second LCD, as well as not obsoleting so much existing hardware. It could also be used with desktop Mac via the Magic Trackpad. This virtual trackpad would be easier to see, and would be adjustable accessibility-wise and localizing-wise.
I was visualizing working with this in my head (I have a knack for that) and I could see that it was amazingly useful to improve interaction.
As a trained artist (among other things) I'm used to having the ability to work with my hands without looking at them -- though it's fairly tough for most people to do. Hence the practice people need with touch typing and not everyone gets it. Having a HUD (Heads Up Display) for the trackpad virtually on the screen would improve people's ability to interact and NOT look at their hands -- and have a better idea spatially of the results of that interaction.
I predict that this will become a "standard" for kiosks and other interaction situations where the User is likely to NOT have a huge amount of experience with the device. Most people take time to get acquainted with keyboards and trackpads with only a slight difference in orientation. Subtle feedback on devices always enhances their accessibility -- and this is one of those GOOD ideas to improve computer interface.I think this is a good idea and I'm surprised there isn't lots of prior art.
Making the touchpad an LCD is bad because 1) your hands cover it. 2) your focus is off the screen and onto the touchpad.
Putting a virtual touchpad on the screen works because our brains can easily do the mapping between the physical device and the screen - like we do with a mouse.
The difficulty with the proposed approach is how much of the screen is covered by the virtual touchpad? You'd have to be very careful to make sure the working area of the virtual touchpad provides valuable controls while not obscuring stuff the user needs to see - think documents in multiple windows and reading stuff in one window needed in another window (instructions, reference material, etc.) Also you'll have to make the target regions the "right" size. Looking at the touchpad on my machine, I'd think 5 regions wide by 4 high might be about right. And switching from mouse/pointer to virtual touchpad would have to be done carefully - what if you have the virtual touchpad open but want to scroll or move a window on the desktop?
Imagine an application like ProTools where an equalizer plug-in could pop-up and map its controls to the touchpad - directly change the EQ curve, or raise and lower sliders using the virtual touchpad. Or in Photoshop a color blender control could pop-up.
I think linear motions (vs turning virtual knobs) and heavy use of the edges and corners would make sense.
This could be a fun technology to have integrated at the OS level - available to apps to use and abuse as they will.
- Jasen.
I see a lot of people saying NO but I really don't see the issue of having a small screen underneath a glass track pad.
Quote:
Originally Posted by Evilution
I see a lot of people saying NO but I really don't see the issue of having a small screen underneath a glass track pad.
You need to have good haptic feedback to make this truly useful. Eventually, something like this will replace the keyboard and the trackpad.
Originally Posted by Evilution
I see a lot of people saying NO but I really don't see the issue of having a small screen underneath a glass track pad.
Boils down to one word: point.