Apple investigating handwriting and selective touch recognition
A pair of patent applications published by the U.S. Patent and Trademark Office on Thursday reveal Apple is investigating the use of heuristics in its mobile devices to deliver a more realistic digital representation of handwriting, as well as selective touch input that ignores extraneous touch events.
Source: USPTO
From the patent application's abstract:
For example, many parcel delivery companies employ a portable signature capture device to facilitate quick and secure service. Usually, the device lacks the processing power to capture enough data points to display a smooth and continuous representation of a customer's handwritten signature.
To overcome this apparent inadequacy found in some devices, Apple's patent leverages a predetermined set of rules to connect the space between two input points by a either a line segment or curve. If the device determines that a curve is needed, a "smoothness adjustment factor" is applied to reduce or maintain smoothing depending on the length between said input points.
Illustration of data points.
Key to the system's functionality is the calculation of velocity and direction, both of which are used in tandem to determine whether a straight line or curve is rendered between two points.
It is important to note that Apple's patent can be implemented for both stylus and finger input, meaning the invention is suitable for devices like the iPad.
There are a number of apps currently on the App Store that take advantage of the iDevice's touch input system to reproduce handwriting, such as Square's credit card system and various drawing apps.
Apple's invention fits perfectly with the handwriting recognition patent described above. As described in the filing's background, when a device is capturing handwritten notes or signatures, a user is apt to rest their palm on the touch sensitive surface for support. This can cause errant touch events, triggering unwanted results.
Example of active region.
From the filing's background:
A number of implementations are described, one of which contains a visual element like the lines of a virtual notebook that corresponds to the underlying active region. In this example, an indicator denotes where writing input can be entered. Once initiated, the writing gesture will continue to be logged until the user interrupts the process by lifting their finger or stylus from the touch sensitive screen. When writing is complete, the displayed input can be saved on the device for later use.
Another example notes that haptic feedback can be used in lieu of visual indicators.
Both of Thursday's patent applications credit Lyndley Crumly as inventor, with David Clark in the handwriting recognition invention, and were first filed in April 2011.
Handwriting Recognition
First filed for in 2011, Apple's "Handwriting capture techniques" describes a set of rules used to render a digital replica of a person's handwriting, or more specifically, how a touch-capable device translates user input into a representation visually similar to handwriting.Source: USPTO
From the patent application's abstract:
The patent looks to solve the problem that arises from collecting and processing a multitude of input data, in this case touch event points, which can be difficult if a stylus or finger is quickly moved across a touch sensitive surface. Tailored for "devices capable of capturing only a few signature data points per second," the invention allows for a more natural representation of a signature or handwritten line when the number of given data points would otherwise be inadequate.A set of rules is used by a processor of a device to render a digital image of handwriting (e.g., handwritten signature) by connecting data points captured on a touch sensitive surface of the device with line segments or curves.
For example, many parcel delivery companies employ a portable signature capture device to facilitate quick and secure service. Usually, the device lacks the processing power to capture enough data points to display a smooth and continuous representation of a customer's handwritten signature.
To overcome this apparent inadequacy found in some devices, Apple's patent leverages a predetermined set of rules to connect the space between two input points by a either a line segment or curve. If the device determines that a curve is needed, a "smoothness adjustment factor" is applied to reduce or maintain smoothing depending on the length between said input points.
Illustration of data points.
Key to the system's functionality is the calculation of velocity and direction, both of which are used in tandem to determine whether a straight line or curve is rendered between two points.
It is important to note that Apple's patent can be implemented for both stylus and finger input, meaning the invention is suitable for devices like the iPad.
There are a number of apps currently on the App Store that take advantage of the iDevice's touch input system to reproduce handwriting, such as Square's credit card system and various drawing apps.
Selective Touch
The second touch-related application published on Thursday was Apple's "Region Activation for Touch Sensitive Surface," which describes a system in which extraneous touch events recognized outside of so-called "active regions" are ignored.Apple's invention fits perfectly with the handwriting recognition patent described above. As described in the filing's background, when a device is capturing handwritten notes or signatures, a user is apt to rest their palm on the touch sensitive surface for support. This can cause errant touch events, triggering unwanted results.
Example of active region.
From the filing's background:
The application suggests that by defining "active regions," inadvertent touch events can be avoided. A region, or regions, can be activated by the user, and any touch event starting in that area is logged and displayed on screen. Touch events not started within the active region or "not associated with the beginning touch event" are ignored.To avoid such inadvertent touch input, the user can elevate their hand above the touch surface when writing. This workaround can be awkward and tiring to the user.
A number of implementations are described, one of which contains a visual element like the lines of a virtual notebook that corresponds to the underlying active region. In this example, an indicator denotes where writing input can be entered. Once initiated, the writing gesture will continue to be logged until the user interrupts the process by lifting their finger or stylus from the touch sensitive screen. When writing is complete, the displayed input can be saved on the device for later use.
Another example notes that haptic feedback can be used in lieu of visual indicators.
Both of Thursday's patent applications credit Lyndley Crumly as inventor, with David Clark in the handwriting recognition invention, and were first filed in April 2011.
Comments
Had anyone conceived of the method for smoothing and correcting handwriting on a tablet? I hadn't. The accidental touch, sure, but the method in which to resolve that certainly not.
Quote:
Originally Posted by SolipsismX
These are the type of refinements that the anti-Apple camp will say are pointless fluff would be quickly followed because it does add to the user experience which would result in them eventually saying it was obvious.
I'm grinding my teeth awaiting the inevitable occurrence of the chain of events as described above. In the meantime my suffering is soothed by wild expectations of future improvements coming from Apple.
Quote:
Originally Posted by SolipsismX
These are the type of refinements that the anti-Apple camp will say are pointless fluff would be quickly followed because it does add to the user experience which would result in them eventually saying it was obvious.
Had anyone conceived of the method for smoothing and correcting handwriting on a tablet? I hadn't. The accidental touch, sure, but the method in which to resolve that certainly not.
Adobe has. This is certainly not revolutionary. Just look at Adobe Ideas on iPad, every stroke is smoothed. Try it!
How's it any different from the Apple camp saying that writing on a touchscreen was pointless fluff, old tech, obsolete, and ridiculing anyone that suggests the iPad come with a stylus?
No one reasonable has ever said there was no case in which a stylus was more ideal. What Jobs said (and reasonable people agreed) was that a stylus on a touchscreen should not be the primary or default way that you interact with the device. Also consider that stylus input on previous devices we almost mostly resistive, not capacitive, which just adds to the pre-iPhone poor user experience of smartphones and PDAs.
I quite like the Samsung Note's integrated Wacom digitizer and their included APIs, and hope that one day it can be added to the iPad because there is a definitely a benefit for this option for a certain class of user or specific usage need... again, not as a primary input for the masses. There is a reason the iPhone is the de facto device in the world and the one that all analysts and vendors have looked to compare and/or best ever since the iPhone was unveiled almost 6 years ago.
Additionally, it's not just foolish to say the iPad should come with a stylus but doubly foolish because 1) it's something that will not be used by most people or often so it should remain an accessory, and 2) until there is a Wacom digitizer the results from any stylus are too simplistic for this "inclusion" you suggest.
Even after the iPad gets a Wacom digitizer built in (which I think is inevitable) I will never, ever, ever, ever want my iPad to be designed around having a stylus that slides into the chassis. Give the HW the proper access with the digitizer; make great APIs that developers can take advantage of; showcase it precision apps like iPhoto; and let vendors sell a variety of stylus to fit all potential needs.
Quote:
Originally Posted by dasanman69
How's it any different from the Apple camp saying that writing on a touchscreen was pointless fluff, old tech, obsolete, and ridiculing anyone that suggests the iPad come with a stylus?
In his defense, (unless my memory is wrong) Solips has indicated his wish of (or has discussed) having a proper Wacom Digitizer on the iPad to use a stylus with and thus extend the iPad's capabilities.
I would love to have a workable stylus with the iPad as well. But the issue I have is a little different. Tapping the iPad screen to select something or doing the various other touch operations like pinch, swipe, etc., are really not that much of a problem. However I find the continuous tapping to write, say in a Pages doc, using the keyboard a little uncomfortable. I think it has to do with not getting a tactile feedback. When I press a key on a physical keyboard, the impact of my finger tapping the key is dampened by the key being pressed. This is not possible in the case of the iPad. So I find it a little uncomfortable.
As a workaround, I do have a bluetooth physical keyboard which I use from time to time, or I simply opt to do my text editing on my iMac/ MBP. So essentially, attanding a workshop/ seminar and trying to take notes on the iPad for me is a little uncomfortable without the external keyboard. If the iPad did have a Digitizer, then I'd simply scribble the notes. Of course, I do have the option of recording the audio of the workshop, but in some cases it just isn't feasible.
Anyway, that was way off-topic, so back to the topic, every patent I read about from Apple seems very interesting. This one falls right into the rubber-banding patent category - to add the finesse to iOS that makes it so good.
Well I didn't mean thee Apple camp, I meant those on here that are Apple loving and everything else hating. Several like yourself have in the past expressed that a stylus would be great to do specific tasks on a iPad only to be shot down with "well if Apple wanted you to use a stylus they would've included one" and have ridiculed Samsung for having devices that come with a stylus. Those very same people will wait on line and gladly overpay for a Apple branded stylus and swear Apple invented the idea.
1) Then those people are what I'd call unreasonable but let's make sure we're clear that we're talking about people that don't think Apple should ever include any type of Wacom digitizer in their touch-based products for the sole reason because they haven't done so already, not simply people that are arguing against it being the de facto input method, boring a hole in the chassis to include one inside each device sold, or any other reasonably conceived position (even if we don't personally agree with it).
2) If Apple makes a great stylus I'd likely buy it. I bought some stylus for my iPad but never use it. I use my iPhone/iPad for work for signatures and fingers simply work better than the foamy capacitance tip of the stylus.
You think anyone is on that R&D?
Quote:
Originally Posted by dasanman69
How's it any different from the Apple camp saying that writing on a touchscreen was pointless fluff, old tech, obsolete, and ridiculing anyone that suggests the iPad come with a stylus?
I would love an iPad with stylus, especially a dual-screen Planner-size one.
Quote:
Originally Posted by ClemyNX
Adobe has. This is certainly not revolutionary. Just look at Adobe Ideas on iPad, every stroke is smoothed. Try it!
This is not that. "Smoothing" has indeed been around for a while but that's just a simple algorithm uniformly applied to any set of points on a line. This is a different thing entirely.
Quote:
Originally Posted by SolipsismX
These are the type of refinements that the anti-Apple camp will say are pointless fluff would be quickly followed because it does add to the user experience which would result in them eventually saying it was obvious.
Had anyone conceived of the method for smoothing and correcting handwriting on a tablet? I hadn't. The accidental touch, sure, but the method in which to resolve that certainly not.
Plenty of Tablets (including those ill-fated Windows tablets that came out circa 2002) have included various forms of handwriting recognition, handwriting to typed text, smoothing, palm/hand recognition, etc...
I know Lenovo tablets (both the X230t and ThinkPad tablet) have it, Adobe has been mentioned, Wacom has it, Palm also had their own version, and I'm sure others.
If Apple does get this patent, no way it sticks - way too much prior art.
Quote:
Originally Posted by SolipsismX
1) Then those people are what I'd call unreasonable but let's make sure we're clear that we're talking about people that don't think Apple should ever include any type of Wacom digitizer in their touch-based products for the sole reason because they haven't done so already, not simply people that are arguing against it being the de facto input method, boring a hole in the chassis to include one inside each device sold, or any other reasonably conceived position (even if we don't personally agree with it).
2) If Apple makes a great stylus I'd likely buy it. I bought some stylus for my iPad but never use it. I use my iPhone/iPad for work for signatures and fingers simply work better than the foamy capacitance tip of the stylus.
I agree. Finger-paintings by famous artists aside, the iPad is just not good for drawing, and until you can use a stylus in the same way as you would in real life, it never will be.
IMO the day you can pull out your iPad, grab a stylus, and do a simple doodle like you would do on a cocktail napkin, is a day when you will see a huge bump in iPad adoption. Humans have been communicating like this since cave-men days. Any universal computing appliance needs this capability.
Quote:
Originally Posted by Mikeb85
Plenty of Tablets (including those ill-fated Windows tablets that came out circa 2002) have included various forms of handwriting recognition, handwriting to typed text, smoothing, palm/hand recognition, etc...
I know Lenovo tablets (both the X230t and ThinkPad tablet) have it, Adobe has been mentioned, Wacom has it, Palm also had their own version, and I'm sure others.
If Apple does get this patent, no way it sticks - way too much prior art.
Your forgetting that a lot of the prior art in that area is actually from the Newton. Newton was made by Apple, and pre-dates everything you mentioned.
Quote:
Originally Posted by Gazoobee
IMO the day you can pull out your iPad, grab a stylus, and do a simple doodle like you would do on a cocktail napkin, is a day when you will see a huge bump in iPad adoption. Humans have been communicating like this since cave-men days. Any universal computing appliance needs this capability.
Spot on. This is exactly what I said a year ago. People who's uneasy of computer will buy device like this en masses.
Apple has surely refer needed there prior patents used in a shipping product at least a far back as Newton.
But you need look at the actual patent not just a rudimentary description. Of your wife is a 5'8" women with medium length black hair and green eyes does that mean she is the same as women fitting that description? Of course not. While this patent may not end up valid is no reason to believe that it's invalid simply because something with handwriting recognition has existed previously.
Originally Posted by ClemyNX
Adobe has. This is certainly not revolutionary. Just look at Adobe Ideas on iPad, every stroke is smoothed. Try it!
I thought of that while I was reading his post, too.
Then I thought for one third of a second longer and realized that every stroke is smoothed… INCORRECTLY.
Quote:
Originally Posted by matrix07
I would love an iPad with stylus, especially a dual-screen Planner-size one.
Dual screen with hinges and kick stand??
Quote:
Originally Posted by Gazoobee
Your forgetting that a lot of the prior art in that area is actually from the Newton. Newton was made by Apple, and pre-dates everything you mentioned.
Actually there are patents for handwriting recognition that pre-date Apple itself, and there were products that incorporated it that pre-dated the Newton.
Quote:
Originally Posted by SolipsismX
Apple has surely refer needed there prior patents used in a shipping product at least a far back as Newton.
But you need look at the actual patent not just a rudimentary description. Of your wife is a 5'8" women with medium length black hair and green eyes does that mean she is the same as women fitting that description? Of course not. While this patent may not end up valid is no reason to believe that it's invalid simply because something with handwriting recognition has existed previously.
No, but odds are with so much prior art (dating back to the 1950's), Apple will have a tough time proving their patent is different enough...
Using a stylus with a computer was one of the very first techniques to interact with a computer, pre-dating the mouse.