iOS 18 Mail app will get huge AI enhancements & summarizations with 'Project Blackpearl'
Apple's Mail application is getting a massive AI upgrade with Project BlackPearl. Here's how iOS 18 will make your life better, and your working day easier.
Project BlackPearl will greatly enhance the Mail app with machine learning and AI
People familiar with the matter have revealed exclusively to AppleInsider that the Mail app will receive substantial improvements through Apple's new AI initiative. In pre-release versions of the company's upcoming operating systems, expected at WWDC, search results within the Mail app include information from Contacts, as well as locations and locally stored documents.
This means that the Mail app will be able to take advantage of Apple's on-device Ajax LLM for more varied search results. This will also allow for so-called "Smart Replies" to emails, generated by Apple's AI software.
Apple's Smart Replies will streamline email communication and will save knowledgeable users a significant amount of time. The feature will be especially useful for business professionals, customer service representatives, or large companies that routinely deal with large amounts of emails.
As detailed in our report on Project Greymatter, Siri and Spotlight will also make use of AI software. Siri will have significantly improved contextual understanding, and will be able to take into account locations, calendar events as well as entities such as people and companies.
Mail, Messages, and Siri will also receive a new "Smart Reply" feature, which will give users the option to reply with an instant AI-generated response to an email or message. Siri will gain the ability to utilize Apple's AI in order to create replies at the user's request.
A customer service representative, tasked with addressing the concerns of various users, would be able to reply to emails much more quickly through Apple's Smart Replies. Administrative assistants could use the feature to set up meetings or send out business-related inquiries.
Pre-release versions of Apple's upcoming operating systems contain a new system-wide writing tools framework as well. People familiar with the new framework have told AppleInsider that it could enable text editing through Siri.
Though the exact implementation of this new framework remains to be seen, there are indications that it would facilitate tone adjustments of user-created text in emails, and elsewhere. Users could instruct Siri to change text with a phrase like: "Make this text more professional," so that their message or email would be taken seriously.
This means that Siri could analyze emails and offer specific suggestions or generate responses to achieve the user's desired effect better. The virtual assistant may provide suggestions that would adjust the overall tone of the user's response or text, making it sound more concise, friendly, or professional.
Project BlackPearl will enable email categorization through machine learning
The Mail application is expected to receive significant enhancements via machine learning, as part of Project BlackPearl. These enhancements will allow for the categorization of emails based on the text contained within them.
Image Credit: Apple
By analyzing the text contents of an email, machine learning software will be able to identify time-sensitive emails and classify email senders as well. People familiar with the matter told us that the software can classify emails into several pre-defined categories:
- Commerce
- News
- Other
- Promotions
- Social
- Transaction
Apple could include these categories in the app's user interface, though it is also possible that this new feature was created to help Siri identify different email types. The iOS 18 and macOS 15 version of Siri will also be able to summarize emails and email threads, among other things.
What else will Siri be able to summarize?
A new-and-improved version of Siri is expected to make its debut at WWDC. Apple's virtual assistant will be able to summarize a wide variety of texts, and now we have some specific information about the text types involved.
Apple's AI is capable of summarizing webpages, similar to Arc Browser
has recently received details about the instructions and prompts used in the development of Apple's generative AI software. The instructions Apple created offer significant insight into the company's plans in the area of artificial intelligence.
Based on information from people familiar with the matter, the generative AI software in iOS 18 and macOS 15 will be able to summarize the following types of text:
- Emails and email threads
- Messages
- Notifications and notification stacks
- Notes
- Posts
- User-provided texts and news articles
Internally, Apple has tested multiple levels of summarization:
- Topic-level summaries - consisting of three topic words, representing the key concepts of a text
- 10 or 20-word summaries, which use clauses rather than full sentences
- Paragraph-level summaries - concise three-sentence paragraph with no more than 60 words
Apple developed multiple summary types to account for specific use cases and UI elements, as certain summaries require more context and specific instructions to achieve the desired result.
More on the different summarization levels and text types
As explained in our exclusive report on Apple's on-device Ajax LLM, the large language model is capable of identifying keywords and sentences within a text. This means that it can identify units of language within a note, message, notification, or webpage, and create short summaries in the form of select keywords.
Apple's Ajax LLM is capable of performing topic-level summarizations entirely on-device
This simple selection of keywords is the basis for the software's topic-level summaries. For this level of summarization, Apple's AI is told to isolate three topic words, representing core concepts from the input text.
Although the Ajax LLM can handle topic-level summaries, advanced summarization, and more complex tasks appear to necessitate cloud-based processing. Apple may have recently partnered with OpenAI to facilitate advanced AI functionality, as indicated by a recent report.
Apart from topic-style summaries, Apple also experimented with 10 to 20-word summarization. With this type of summary, Apple's generative AI software is told to use clauses rather than complete sentences and to generate a result with no more than 10 or 20 words.
Paragraph-level summarization requires Apple's AI software to create a three-sentence concise paragraph consisting of no more than 60 words. Specific instructions are given based on the type of text the software is meant to summarize.
How does Apple instruct and guide its artificial intelligence?
Although Apple's approach to AI and LLMs does not include a chatbot, instructions used for training the software bear some resemblance to ChatGPT-style prompts. Apple has reportedly secured a licensing deal with OpenAI, meaning that we could see OpenAI's technology in Apple's new operating systems.
Apple has created ChatGPT-style prompts for its AI
The company's instructions begin by directly stating that the AI is to assume the role of an expert in summarizing a specific type of text. The AI is told to maintain the role, or else its responses will not be considered useful or helpful - indicating that the overall goal is to assist the end users of Apple's operating systems.
When summarizing a stack of notifications, the AI has to focus on the common topic of all notifications within said stack. Similarly, when analyzing and creating a summary of user-supplied text such as an article, the software is instructed to focus on the main topic of the text in its entirety.
For texts provided by the end-user, the AI is instructed to refrain from using unnecessary prefaces and to exclude minor details. To generate a summary of recent messages or notification stacks, the software must put emphasis on key information - the names of the sender and recipient as well as the time and date.
In essence, this all means that Apple's virtual assistant - Siri - will be the main recipient of the company's LLM-powered summarization upgrades. Through generative AI and integration with core system applications, Siri will be able to do much more than ever before.
What else will Siri be able to do?
According to people familiar with the matter, Siri is also expected to receive significantly enhanced in-app functionality. This means that Apple's virtual assistant will be able to perform arguably complex tasks within various system applications.
Siri will gain the ability to perform in-app tasks in apps such as Photos, Messages and Notes
In Photos, for instance, users will have the option to enhance individual photos, open specific types of images, play memories, and set their device wallpaper through Apple's virtual assistant.
Users could say something like "Enhance this photo" or "Show me photos from last year" to facilitate these actions. Siri will likely be able to display Photos from the user's library rather than immediately searching the web for images.
Similar Siri functionality is expected to become available within other system applications such as Messages and the built-in Notes app. The virtual assistant will be able to move individual notes, open and analyze documents, and send images or files via the Messages app.
The default Photos app will also receive some minor UI changes, as certain interface elements will be reorganized or renamed. The design change itself is little more than a stylistic choice, though, since other AI-powered editing features are the main update we can look forward to.
Generate and edit images with Apple's AI software
Apple's AI software is also capable of generating images in internal test environments. Now we have more details as to the different types of images the software can create.
Apple's built-in Photos app will receive a new Clean Up tool, along with some UI changes
According to people familiar with the matter, the company's pre-release operating systems feature an entirely new image generation framework. With it, it's possible to create different types of images by employing artificial intelligence.
The new image generation framework references the following image types and styles:
- Animation
- Emoji
- Illustration
- Line Art
- Sketch
As Apple has developed its own internal image generation app, along with a new framework for it - it's likely that the company will make this feature available to end users in some form. One report indicated that AI-generated emojis are in the works, and our findings corroborate this claim.
AI-powered image editing is also in the works, as Apple intends to replace its existing Retouch Tool with a new feature called Clean Up. The Clean Up option will allow users to remove objects from photos through the use of generative AI.
It's worth emphasizing, however, that the information comes from people familiar with pre-release versions of Apple's operating systems and test environments. Apple has been known to cancel, delay or rename new OS features ahead of release, so there's always a chance the features mentioned here won't become available to the general public.
Apple is set to debut its next-gen operating systems, such as iOS 18, iPadOS 18 and macOS 15, at its annual Worldwide Developers' Conference (WWDC) on June 10. At the event, we'll likely see a revamped Calculator along with upgraded versions of the built-in Calendar, Notes and Safari applications.
Read on AppleInsider
Comments
I want 2 things:
1. graybars
2. the ability to change the subject of a received email (a là Eudora)
Today simple transcription is barely at 60% accurate for me. I have to scream at my HomePod in the morning to get it to play the Economist Intelligence podcast, even though I ask Siri to do this every single day right after I say "Hey Sire, what is my update?"
As I noted in the WSJ, during Apple's last earnings report the live screen transcription was a mess mis-stating what Tim had just stated. Embarrassing to say the least.
And fwiw, I use flags for completely different things.
Hopefully Apple will provide switches for users to turn off any and all AI “features.”
maybe Johnny depp can be the new “Clippy!”
- ordering and delivery confirmations
- status updates
- (secondary authentication) login codes
- email lists
Actually writing mails? A couple every week.