Apple talks about the origins of Math Notes in iPadOS 18

AppleInsiderAppleInsider
Posted:
in iPadOS edited July 17

In a new interview, Apple's Jenny Chen and Ty Jordan have revealed how both the new Math Notes and Smart Script for iPadOS 18 have their roots in years of Apple technology.

Handwritten math equations and solutions on a tablet screen titled Scratch Math, including arithmetic and trigonometric problems.
Math Notes on an iPad



Math Notes is the new feature where iPadOS 18 will not only recognize a handwritten sum or equation, but it will solve it -- and solve it in the user's own handwriting. Then with Smart Script, your actual handwriting gets smoothed out intelligently, so that it preserves what makes your writing yours, but also makes it more legible.

In a new YouTube interview, engineering manager Jenny Chan, and product manager Ty Jordan, say that these features are the culmination of what else Apple has been working on for a long time.



"So this actually builds on a lot of the technology that we've been building over many years," said Chen. "So with Scribble [handwriting recognition], we spent a lot of time actually working on recognizing [text]... but, basically, we were able to recognize your handwriting, turn it into typed text."

"And so I feel like [Math Notes is] just taking the technology from Scribble and taking it to the next level of not only recognizing, but being able to also synthesize text that's actually similar to your handwriting," she continued.

Ty Jordan said that the ability to recognize text and then emulate it also gave Apple the ability to manipulate handwriting and give it some of the features of typewritten text.

"So if you tap and hold your pencil, it can just move words around and you'll see that reflows in your paragraph so you can get some more space," he said. "You can even just paste typed text as your own handwritten text right in the middle of a paragraph."

"So a lot of the things that we take for granted with typed text," he continued, "you can now do with hand text, which is really cool."

Math Notes and Smart Script are part of iPadOS 18, which will be officially launched in the fall. A public beta has just been released, and Apple is now several rounds into beta testing with developers.



Read on AppleInsider

Comments

  • Reply 1 of 7
    brianobriano Posts: 1member
    Interesting Video - but just how many times can you say 'Awesome' and 'literally'
    I guess none of these people were around when the Apple Newton was under development in 1987...
    :-)
    nubuswatto_cobra
  • Reply 2 of 7
    Stabitha_ChristieStabitha_Christie Posts: 528member
    briano said:
    Interesting Video - but just how many times can you say 'Awesome' and 'literally'
    I guess none of these people were around when the Apple Newton was under development in 1987...
    :-)
    Obviously not around when the Newton was in development. The key difference is that their stuff actually works as intended. The Newton handwriting recognition sucked diddly ucked until it was repurposed as Inkwell in 2002. RIP Inkwell. 
    watto_cobra
  • Reply 3 of 7
    nubusnubus Posts: 544member
    Stabitha_Christie said:
    briano said:
    Interesting Video - but just how many times can you say 'Awesome' and 'literally'
    I guess none of these people were around when the Apple Newton was under development in 1987...
    :-)
    Obviously not around when the Newton was in development. The key difference is that their stuff actually works as intended. The Newton handwriting recognition sucked diddly ucked until it was repurposed as Inkwell in 2002. RIP Inkwell. 
    The data soup of NewtOS was interesting but impossible to sync to other devices, paying extra for TCP/IP-support was mad, but people wanted to touch and try it.

    I really like what Apple is doing here. Not sure any other company would be both interested in and able to.
    watto_cobra
  • Reply 4 of 7
    Stabitha_ChristieStabitha_Christie Posts: 528member
    nubus said:
    Stabitha_Christie said:
    briano said:
    Interesting Video - but just how many times can you say 'Awesome' and 'literally'
    I guess none of these people were around when the Apple Newton was under development in 1987...
    :-)
    Obviously not around when the Newton was in development. The key difference is that their stuff actually works as intended. The Newton handwriting recognition sucked diddly ucked until it was repurposed as Inkwell in 2002. RIP Inkwell. 
    The data soup of NewtOS was interesting but impossible to sync to other devices, paying extra for TCP/IP-support was mad, but people wanted to touch and try it.

    I really like what Apple is doing here. Not sure any other company would be both interested in and able to.
    I had a Newton 120 and it was fun to play with but I never found a "must have" use for it. It was ahead of its time and probably shouldn't have been released in the state that it was in. My understanding was that the MP2000 and MP2100 were both far better. 
    watto_cobra
  • Reply 5 of 7
    thttht Posts: 5,585member
    Stabitha_Christie said:
    nubus said:
    Stabitha_Christie said:
    briano said:
    Interesting Video - but just how many times can you say 'Awesome' and 'literally'
    I guess none of these people were around when the Apple Newton was under development in 1987...
    :-)
    Obviously not around when the Newton was in development. The key difference is that their stuff actually works as intended. The Newton handwriting recognition sucked diddly ucked until it was repurposed as Inkwell in 2002. RIP Inkwell. 
    The data soup of NewtOS was interesting but impossible to sync to other devices, paying extra for TCP/IP-support was mad, but people wanted to touch and try it.

    I really like what Apple is doing here. Not sure any other company would be both interested in and able to.
    I had a Newton 120 and it was fun to play with but I never found a "must have" use for it. It was ahead of its time and probably shouldn't have been released in the state that it was in. My understanding was that the MP2000 and MP2100 were both far better. 
    It was impossible for Apple achieve the proper form factor with the Newton. Display too small, device too thick. Palm solved for half the problem by simplifying the UI and making it a pocketable handheld, but they gave up the intriguing part of a digital notebook where you can handwrite on it. Just took another 20 years to get there with iOS 18 and machine learning models.

    Yes, like the Apple engineers in the video, I would highly encourage handwritten notes in school. Typing notes just isn't the same thing. The act of writing down equations really helps you understand the math. Doing this in Latex is cumbersome and I'm not sure there is retention with it.
    watto_cobra
  • Reply 6 of 7
    dewmedewme Posts: 5,614member
    I’d like to see Apple make a version of this that runs live in Pages and Keynote on iPadOS. I am pretty sure you can copy and paste from Math Notes images into any word processor or presentation app, but having math notes embedded live in other apps would be very useful for putting together engineering documents, keeping an engineering journal, white papers, reports that present formulas with live data, etc. Having the math notes embedded would ensure that everything in your document or presentation stays synchronized. 

    Math Notes looks like a decent educational tool. It’s not yet at a level of apps like Math CAD, but it’s a good first step. Will I use it? Probably not much because my handwriting is pretty bad and I no longer require much more than a basic scientific calculator. If I was showing someone how I came up with a result of some sort this app may be an excellent fit. I’ll take it for a spin soon. 
    watto_cobra
  • Reply 7 of 7
    ChrisLChrisL Posts: 1member
    Why is it that no one speaks about "Tydlig" for iPad which did similar things way back: http://tydligapp.com.
    Not sure if the app is still available but I have been using it regularly over the years and it was quite a novelty in its approach at the time.
    watto_cobra
Sign In or Register to comment.