iOS 18 Project Greymatter will use AI to summarize notifications, articles and much more

Posted:
in iOS

Apple's next-gen operating systems will feature Project Greymatter, bringing a multitude of AI-related enhancements. We have new details on AI features planned for Siri, Notes and Messages.

Siri icon next to Safari and Messages app icons, with 'Spotlight Search' text in the search bar.
AI will improve several core apps with summarization and transcription features



Following widespread claims and reports about AI-related enhancements in iOS 18, AppleInsider has received additional information about Apple's plans in the area of AI.

People familiar with the matter have revealed that the company is internally testing a variety of new AI-related features ahead of its annual WWDC. Known under project codename "Greymatter," the company's AI improvements will focus on practical benefits for the end user.

In pre-release versions of Apple's operating systems, the company has been working on a notification summarization feature known as "Greymatter Catch Up." The feature is tied to Siri, meaning that users will be able to request and receive an overview of their recent notifications through the virtual assistant.

Siri is expected to receive significantly updated response generation capabilities, through a new smart response framework, as well as Apple's on-device LLM. When generating replies and summaries, Siri will be able to take into account entities such as people and companies, calendar events, locations, dates, and much more.



In our earlier reports on Safari 18, the Ajax LLM, and the updated Voice Memos app, AppleInsider revealed that Apple plans to introduce AI-powered text summarization and transcription to its built-in applications. We have since learned that the company intends to bring these features to Siri as well.

This ultimately means that Siri will be able to answer queries on-device, create summaries of lengthy articles, or transcribe audio as in the updated Notes or Voice Memos applications. This would all be done through the use of the Ajax LLM or cloud-based processing for more complex tasks.

We were also told that Apple has been testing enhanced and "more natural" voices, along with text-to-speech improvements, which should ultimately result in a significantly better user experience.

Apple has also been working on cross-device media and TV controls for Siri. This feature would allow someone to, for instance, use Siri on their Apple Watch to play music on another device, though the feature is not expected until later in 2024.

The company has decided to embed artificial intelligence into several of its core system applications, with different use cases and tasks in mind. One notable area of improvement has to do with photo editing.

Apple has developed generative AI software for improved image editing



iOS 18 and macOS 15 are expected to bring AI-powered photo editing options to applications such as Photos. Internally, Apple has developed a new Clean Up feature, which will allow users to remove objects from images through the use of generative AI software.

A small dog wearing a polka dot bandana is on a leash, facing away in a fenced yard.
The Clean Up tool will replace Apple's current Retouch tool



Also related to Project Greymatter, the company has created an internal-use application known under the name "Generative Playground." People familiar with the application revealed exclusively to AppleInsider that it can utilize Apple's generative AI software to create and edit images, and that it features iMessage integration in the form of a dedicated app extension.

In Apple's test environments, it's possible to generate an image through artificial intelligence and then send it off through iMessage. There are indications that the company has a similar feature planned for the end users of its operating systems.

This information aligns with another report claiming that users will be able to use AI to generate unique emojis, though there are additional possibilities for image-generation features.

According to people familiar with the matter, pre-release versions of Apple's Notes application also contain references to a Generation Tool, though it is unclear if this tool will generate text or images - as is the case with the Generative Playground app.

Notes will receive AI-powered transcription and summarization, along with Math Notes



Apple has prepared significant enhancements for its built-in Notes application, set to make its debut with iOS 18 and macOS 15. The updated Notes will receive support for in-app audio recording, audio transcription, and LLM-powered summarization.

iPhone screen showing Voice Notes recording feature and the text 'Voice Notes iOS 18' on a gray background.
iOS 18's Notes app will support in-app audio recording, transcription and summarization



Audio recordings, transcriptions, and text-based summaries will all be available within one note, alongside any other material users choose to add. This means that a single note could, for instance, contain a recording of an entire lecture or meeting, complete with whiteboard pictures and text.

These features would turn Notes into a true powerhouse, making it the go-to app for students and business professionals alike. Adding audio-transcription and summarization features will also allow Apple's Notes application to better position itself against rival offerings such as Microsoft's OneNote or Otter.

While support for audio recording at the application level, along with AI-powered audio transcription and summarization features will all greatly improve the Notes app - they're not the only things Apple has been working on.

Math Notes -- create graphs and solve equations through the use of AI



The Notes app will receive an entirely new addition in the form of Math Notes, which will usher in support for proper mathematical notation and enable integration with Apple's new GreyParrot Calculator app. We now have additional details as to what Math Notes will entail.

Two smartphones display notes and calculator apps with a yellow icon in the center. The first screen shows an audio recording interface; the second shows a calculator with the number 64.
iOS 18's Notes app will introduce support for AI-assisted audio transcription and Math Notes



People familiar with the new feature have revealed that Math Notes will allow the app to recognize text in the form of mathematical equations and offer solutions to them. Support for graphing expressions is also in the works, meaning we could see something similar to the Grapher app on macOS, but within Notes.

Apple is also working on input-focused enhancements related to mathematics, in the form of a feature known as "Keyboard Math Predictions." AppleInsider was told that the feature would enable the completion of mathematical expressions whenever they're recognized as part text input.

This means that, within Notes, users would receive an option to auto-complete their mathematical equations in a similar way to how Apple currently offers predictive text or inline completions on iOS - which are also expected to make their way to visionOS later this year.

Apple's visionOS will also see improved integration with Apple's Transformer LM - the predictive text model which offers suggestions as you type. The operating system is also expected to receive a redesigned Voice Commands UI, which serves as an indicator of just how much Apple values input-related enhancements.

The company is also looking to improve user input through the use of so-called "smart replies," which will be available in Messages, Mail, and Siri. This would allow users to reply to messages or emails with basic text-based responses generated instantly by Apple's on-device Ajax LLM.

Apple's AI versus Google Gemini and other third-party products



AI has made its way to virtually every application and device. The use of AI-focused products such as OpenAI's ChatGPT and Google Gemini have seen a significant increase in overall popularity as well.

Blue and white lines converge at the center with a diamond shape above and the word Gemini below.
Google Gemini is a popular AI Tool



While Apple has developed its own AI software to better position itself against the competition, the company's AI is not nearly as impressive as something like Google Gemini Advanced, AppleInsider has learned.

During its annual Google I/O developer conference on May 14, Google showcased an interesting use case for artificial intelligence - where users could ask a question in video form and receive an AI-generated response or suggestion.

As part of the event, Google's AI was shown a video of a broken record player and asked why it wasn't working. The software identified the model of the record player and suggested that the record player may not be properly balanced and that it did not work because of this.

The company also announced Google Veo - software that can generate video through the use of artificial intelligence. OpenAI also has its own video-generating model known as Sora.

Apple's Project Greymatter and Ajax LLM cannot generate or process video, meaning that the company's software cannot answer complex video questions about consumer products. This is likely why Apple sought to partner with companies such as Google and OpenAI to reach a licensing agreement and make more features available to its user base.

A bright orange digital gadget with a screen displaying the time 1:09 and battery at 98%, alongside two buttons on the side.
Apple will compete with products like the Rabbit R1 by offering vertically integrated AI software on established hardware



Relative to physical AI-themed products, such as the Humane AI Pin or Rabbit R1, Apple's AI projects have a significant advantage in that they run on devices users already own. This means that users will not have to purchase a special AI device to enjoy the benefits of artificial intelligence.

Humane's AI Pin and the Rabbit R1 are also commonly regarded as unfinished or partly functional products, and the latter was even revealed to be little more than a custom Android application.

Apple's AI-related projects are expected to make their debut at the company's annual WWDC on June 10, as part of iOS 18 and macOS 15. Updates to the Calendar, Freeform, and System Settings applications are also in the works.



Read on AppleInsider

hecalder

Comments

  • Reply 1 of 8
    ssfe11ssfe11 Posts: 40member
    Appleinsider coming up with the Apple AI goods. Nice job!
    watto_cobra
  • Reply 2 of 8
    tipootipoo Posts: 1,147member
    I hope there's AI features to end the massive influx of spam calls I've been getting. I believe this is a thing on Android. Send to Voicemail with the text readout somewhat helps but still getting the calls is annoying. 
    lam92103williamlondonwatto_cobra
  • Reply 3 of 8
    hecalderhecalder Posts: 13member
    Sounds like Siri will catch up to an equivalent of ChatGPT 3.5 but with current information. It’s in the steps for a long due improvement but hopefully from there on they will continue to improve much faster to make Siri as good as 4 Omni by the end of the year. Even if the rest advance further but at least I will feel motivated to use Siri because right now I rarely engage with Siri at all. 
    williamlondonwatto_cobra
  • Reply 4 of 8
    CheeseFreezeCheeseFreeze Posts: 1,281member
    It sounds like they will incorporate some sort of a SLM (short language model) that is lightweight and can run on-device for several systems services. 
    For example, just like copy.paste, there will be ‘summarize’, ‘correct’ or what-have-you.
    Which I hope will be part of a better designed UI component for text editing. Right now when I select text on my phone I’m given so many options that I sometimes need to tap 5 times to find the right option.
    Which is a challenging component to improve since it needs to stay out of the way of editing and can only take up a little bit of space. 
    hecalder said:
    Sounds like Siri will catch up to an equivalent of ChatGPT 3.5 but with current information. It’s in the steps for a long due improvement but hopefully from there on they will continue to improve much faster to make Siri as good as 4 Omni by the end of the year. Even if the rest advance further but at least I will feel motivated to use Siri because right now I rarely engage with Siri at all. 
    My guess is that they have a multi-layered process: 

    1. try to complete the request on-device with a simple LLM/SLM
    2. If the request is outside the scope of on-device capabilities, send to the Cloud (which might be the OpenAI partnership that is rumored, and Apple most likely cannot compete with).

    For some functions (1) without (2) might suffice. It is actually very hard to detect whether a request is better off handled by the Cloud instead because a language model will always try to help the user despite the outcome. As a non-deterministic concept it cannot self-assess its output quality that easily.

     
    edited May 30 watto_cobra
  • Reply 5 of 8
    DAalsethDAalseth Posts: 2,829member
    This means that, within Notes, users would receive an option to auto-complete their mathematical equations in a similar way to how Apple currently offers predictive text or inline completions on iOS
    Oh great. That is a function that I turned off after about ten seconds. I hate, no DESPISE predictive text. I kill it wit fire whenever that abomination appears. 
    williamlondon
  • Reply 6 of 8
    Gary-GGary-G Posts: 3unconfirmed, member
    To run even the small AI models they will need to increase the RAM of the I-phone. My guess, 12 GB will be needed to run on device AI properly, or more, perhaps 16 GB if you want the language model to seamlessly integrate in real time with applications that are not native. The AI will be limited at first to machine learning and predictive text, which exists already on the device. The more advanced features will allow Language Models to advance Siri performance for those interested in this feature. I don't agree with on phone mathematical programs that 'solve' equations. This will critically damage the ability of teachers to teach students in middle school, grade school, and college. The level of distraction is still very high, adding more distraction on device and then allowing the device to effectively cheat for the students will overtime decrease the motivation of large classes of students, who need to learn those basic skills for critical thinking and on the job performance. AI is turning out to be a gold rush to sell devices and breath new life into market segments that have plateaued. The question whether all this push technology adds up to any of the real pull technology advances, like messaging, that are already on the device, or is it just 'must have' novelty, which it seems it is shaping up to be. 
    williamlondon
  • Reply 7 of 8
    dewmedewme Posts: 5,476member
    I hope that Apple can scale their AI implementation in a way that doesn't take an "all or nothing" approach when it comes to older Apple devices. The scaling would certainly have a hard limit to the point of having no local AI capability on some older devices. However it would be very nice if newer generation devices like iPhone 13 -> iPhone 15 (along with similar generation iPads and Macs) could have the ability to provide some scaled-back AI capabilities using the hardware capabilities of these older devices, even if it's limited to Pro/Pro Max variants. 

    The last thing I want to see is Apple taking the same approach that Microsoft is taking by requiring substantial upgrades to even very recent hardware platforms to fully enjoy the anticipated benefits of AI. 

    I'm still waiting to see exactly what Apple can deliver under the AI umbrella, which seems more like a circus tent at times, but if it makes my life better, easier, frees me from repetitive and mundane tasks, or picks up the slack when I'm at risk of falling off the rails, I'm all in. I think the jury is still out when it comes to exactly how AI will personally impact our everyday lives. Huge potential, but what will it really deliver to ordinary users? 

    Amazon has been delivering AI based summaries of user reviews in their store for some time. They clearly identify these summaries as being AI generated, which is nice. Their AI summaries mostly roll-up commonly mentioned attributes that appear in user reviews. Do these summaries benefit my purchase decision process? Maybe a little bit because most of the attributes that are pulled out, say performance, have positive and negative counts. The AI summaries do provide a finer grained assessment of specific product qualities compared to the star ratings alone, which are an aggregate of all plusses and minuses at each star rating. This may save you some time digging into the individual reviews, for example the 1-star reviews, to see what each reviewer didn't like about the product. Without pulling out individual attributes you had to read through multiple reviews to determine whether the issues mentioned are things that matter to you.

    Amazon is obviously doing all of this on their end, not on the device. But to be perfectly clear, the quality of the summaries is heavily dependent of the quality of the inputs that drive the summarization process. What about fake reviews, seeded reviews. or AI generated reviews? Hmmm.

    I really hope that all people who are impacted by AI generated content learn and never forget the meaning of "garbage in - garbage out."  A keen awareness of garbage in - garbage out is now a fundamental survival skill that everyone will need to have in order to move forward, not only for buying crap, but for most of what impacts us every minute of every day.
    edited June 2 muthuk_vanalingam
  • Reply 8 of 8
    Sure hope we can turn most, if not all of the ai stuff off. Fine to have it there but I’d like the option to not use it.
    williamlondon
Sign In or Register to comment.