Future iPad or iPhone could both recognize and display handwriting

Posted:
in General Discussion edited July 2020
Apple is continuing to investigate handwriting recognition for touch screens, but the latest research also covers displaying handwriting on an iPhone or iPad screen.

As well as recognizing handwriting, future devices could display text in the same way
As well as recognizing handwriting, future devices could display text in the same way


Apple has regularly filed patents regarding different aspects of handwriting recognition, but a new one is equally concerned with both inputting handwriting and displaying it, too. As well as actually recognizing and reacting to a user writing on the screen, the device may respond with simulated text instead of the usual typewritten one.

The aim is to make using computing devices more natural, and specifically to cut down on the number of mouse or trackpad clicks and taps for selecting and manipulating objects on the screen.

"[Traditional methods] for performing these manipulations are cumbersome and inefficient," says Apple in "Device, Method, and Graphical User Interface for Simulating and Interacting with Handwritten Text."

"For example," it continues, "using a sequence of mouse based inputs to select one or more user interface objects and perform one or more actions on the selected user interface objects is tedious and creates a significant cognitive burden on a user. In addition, these methods take longer than necessary, thereby wasting energy. This latter consideration is particularly important in battery-operated devices."

In outlining the patent's claims, Apple describes the sequence of events that a user will see.

"A device receives a user input that corresponds with a sequence of characters," it begins. "In response to the user input, the device displays simulated handwritten text that includes varying the appearance of characters in the simulated handwritten text based on variations in handwritten text of a respective user."

The detail of the patent is concerned with how a system can respond to different pen strokes, including line drawings, and variations on how characters are written.

These variations are not only recognized, but used as the basis for how the device's displayed handwriting varies in return.

"In response to receiving the user input, the method includes displaying, on the display simulated handwritten text that includes varying the appearance of characters in the simulated handwritten text based on variations that were detected in handwritten text of a respective user," continues the patent.

Detail from patent showing a stylus writing on an iPad
Detail from patent showing a stylus writing on an iPad


Beyond shape recognition, the patent discusses how pressure can be monitored.

"For example, one or more force sensors underneath or adjacent to the touch-sensitive surface are, optionally, used to measure force at various points on the touch-sensitive surface," it says. "Similarly, a pressure-sensitive tip of a stylus is, optionally, used to determine a pressure of the stylus on the touch-sensitive surface."

The two inventors listed on the patent are Daniel E. Gobera Rubalcava and Ryan S Dixon.

Dixon has very many previous patents, including several that are relevant to this one, such as "Managing real-time handwriting recognition," US Patent No. 10,346,035.

It follows prior Apple patents on the topic including one from 2011 about selective touch recognition, and a 2015 filing about continuous handwriting input.

Comments

  • Reply 1 of 17
    DAalsethDAalseth Posts: 2,216member
    That would be pretty amazing.
    Half the time I can't read my OWN handwriting.
    razorpitwatto_cobra
  • Reply 2 of 17
    How does this technology compare and contrast with Inkwell (aka “Ink”) that was active in macOS up until last October? It didn’t get converted to 64-bit so it went bye-bye. I have always wondered why Ink didn’t get ported to iPad Pro / Apple Pencil. 
    edited January 2020 dysamoriawatto_cobra
  • Reply 3 of 17
    Revamp of the Newton handwriting technology from the 1990s. It's about time!
    razorpitdysamoriawatto_cobra
  • Reply 4 of 17
    razorpitrazorpit Posts: 1,796member
    andyring said:
    Revamp of the Newton handwriting technology from the 1990s. It's about time!
    I was a Palm user back in the mid 90's. I've been disappointed for years that the development of handwriting technology not only stopped, but pretty much disappeared. With the raw processing power we have today I bet Apple could really knock it out of the park.
    dysamoriawatto_cobra
  • Reply 5 of 17
    maltzmaltz Posts: 340member
    "Beat up Martin"
    hexclockmattinozwatto_cobra
  • Reply 6 of 17
    MacProMacPro Posts: 19,500member
    I've all but lost the ability to handwrite I've typed for so long.
    edited January 2020 montrosemacsRayz2016watto_cobra
  • Reply 7 of 17
    Apple can’t even correct spelling errors when typing.  I frequently have to start typing in Safari (which uses Google) to correct misspelled words.

    As I supposed to think Apple is going to understand my handwriting?  I’ve typed 99.95% of everything since high school... don’t make me laugh.

    How about Apple purchase Grammarly, then another company with proven handwriting recognition software.


    montrosemacs
  • Reply 8 of 17
    pscooter63pscooter63 Posts: 1,064member
    Not quite the same thing, but there are a few current tablet apps that interpret hand-drawn inputs... for example, Notion allows a user to draw musical notation in a detection box, and extrapolates it to legal input.  But it's a little slow, and it's tricky to reverse changes if the app gets its interpretation wrong.
    watto_cobra
  • Reply 9 of 17
    There’s already some kind of (very rudimentary) handwriting recognition built into Notes. Try creating a drawing in the app and hand writing the word “handwriting“. If you then close the note, you’ll find “handwriting“ is then the title of the note. The app has read your handwriting. 
    edited January 2020 hexclockfastasleepdysamoriawatto_cobra
  • Reply 10 of 17
    hexclockhexclock Posts: 1,072member
    maltz said:
    "Beat up Martin"
    "Eat up Martha"

    watto_cobra
  • Reply 11 of 17
    entropysentropys Posts: 3,606member
    I’m using Selvy PenScript for a universal keyboard at the moment.
  • Reply 12 of 17
    hexclockhexclock Posts: 1,072member

    kayess said:
    There’s already some kind of (very rudimentary) handwriting recognition built into Notes. Try creating a drawing in the app and hand writing the word “handwriting“. If you then close the note, you’ll find “handwriting“ is then the title of the note. The app has read your handwriting. 
    Confirmed. Pretty cool!
    watto_cobra
  • Reply 13 of 17
    Rayz2016Rayz2016 Posts: 6,957member
    And handwriting recognition is already built into the Apple Watch (sort of)
    watto_cobra
  • Reply 14 of 17
    mattinozmattinoz Posts: 1,913member
    Apple can’t even correct spelling errors when typing.  I frequently have to start typing in Safari (which uses Google) to correct misspelled words.

    As I supposed to think Apple is going to understand my handwriting?  I’ve typed 99.95% of everything since high school... don’t make me laugh.

    How about Apple purchase Grammarly, then another company with proven handwriting recognition software.


    There seems to also be instances you have dueling spell checkers acting at the same time on the same text input. The issue is far worse on some social platforms than others.

    dysamoriawatto_cobra
  • Reply 15 of 17
    Now that we have the Apple Pencil, I strongly want handwriting recognition that can go into multiple apps. Perhaps I'm too old school, or note-taking, especially during interviews or meetings, typing just doesn't cut it for me. I had (and still have) the MessagePad 100 and 2100 models. I *really* miss the ability to jot my notes. I use Notabilty on my iPad Pro but I want it everywhere. Its just a natural form of input. 
    dysamoriawatto_cobra
  • Reply 16 of 17
    I use GoodNotes on my iPad Pro. It’s awesome as if the handwriting recognition.
    watto_cobra
  • Reply 17 of 17
    dysamoriadysamoria Posts: 3,430member
    It’s a shame that post-Jobs-pre-Jobs-again Apple pushed the Newton out before the handwriting recognition was usable. By the last model and last version, it worked.

    Too bad the hardware was clumsy. Still is. Apple still haven’t learned to use something other than hard plastic against glass for stylus tips :grimace:
Sign In or Register to comment.