iOS 18 Mail app will get huge AI enhancements & summarizations with 'Project Blackpearl'

Posted:
in iOS

Apple's Mail application is getting a massive AI upgrade with Project BlackPearl. Here's how iOS 18 will make your life better, and your working day easier.

AI project named Project BlackPearl, displayed next to icons for mail and Siri, offering AI upgrades for iOS 18 and macOS 15, on a blue gradient background.
Project BlackPearl will greatly enhance the Mail app with machine learning and AI



People familiar with the matter have revealed exclusively to AppleInsider that the Mail app will receive substantial improvements through Apple's new AI initiative. In pre-release versions of the company's upcoming operating systems, expected at WWDC, search results within the Mail app include information from Contacts, as well as locations and locally stored documents.

This means that the Mail app will be able to take advantage of Apple's on-device Ajax LLM for more varied search results. This will also allow for so-called "Smart Replies" to emails, generated by Apple's AI software.

Apple's Smart Replies will streamline email communication and will save knowledgeable users a significant amount of time. The feature will be especially useful for business professionals, customer service representatives, or large companies that routinely deal with large amounts of emails.

As detailed in our report on Project Greymatter, Siri and Spotlight will also make use of AI software. Siri will have significantly improved contextual understanding, and will be able to take into account locations, calendar events as well as entities such as people and companies.

Mail, Messages, and Siri will also receive a new "Smart Reply" feature, which will give users the option to reply with an instant AI-generated response to an email or message. Siri will gain the ability to utilize Apple's AI in order to create replies at the user's request.

A customer service representative, tasked with addressing the concerns of various users, would be able to reply to emails much more quickly through Apple's Smart Replies. Administrative assistants could use the feature to set up meetings or send out business-related inquiries.

Pre-release versions of Apple's upcoming operating systems contain a new system-wide writing tools framework as well. People familiar with the new framework have told AppleInsider that it could enable text editing through Siri.

Though the exact implementation of this new framework remains to be seen, there are indications that it would facilitate tone adjustments of user-created text in emails, and elsewhere. Users could instruct Siri to change text with a phrase like: "Make this text more professional," so that their message or email would be taken seriously.

This means that Siri could analyze emails and offer specific suggestions or generate responses to achieve the user's desired effect better. The virtual assistant may provide suggestions that would adjust the overall tone of the user's response or text, making it sound more concise, friendly, or professional.

Project BlackPearl will enable email categorization through machine learning



The Mail application is expected to receive significant enhancements via machine learning, as part of Project BlackPearl. These enhancements will allow for the categorization of emails based on the text contained within them.

White envelope icon on a blue gradient background, featuring 'Apple Park, California 95014' text on the envelope flap.
Image Credit: Apple



By analyzing the text contents of an email, machine learning software will be able to identify time-sensitive emails and classify email senders as well. People familiar with the matter told us that the software can classify emails into several pre-defined categories:


  • Commerce

  • News

  • Other

  • Promotions

  • Social

  • Transaction



Apple could include these categories in the app's user interface, though it is also possible that this new feature was created to help Siri identify different email types. The iOS 18 and macOS 15 version of Siri will also be able to summarize emails and email threads, among other things.

What else will Siri be able to summarize?



A new-and-improved version of Siri is expected to make its debut at WWDC. Apple's virtual assistant will be able to summarize a wide variety of texts, and now we have some specific information about the text types involved.

Two smartphones showing an article about iOS 18 updates, including AI advancements, new look, redesigned stock apps, and more major redesigns for Notes, Mail, Fitness, Photos, and Calculator.
Apple's AI is capable of summarizing webpages, similar to Arc Browser
AppleInsider

has recently received details about the instructions and prompts used in the development of Apple's generative AI software. The instructions Apple created offer significant insight into the company's plans in the area of artificial intelligence.

Based on information from people familiar with the matter, the generative AI software in iOS 18 and macOS 15 will be able to summarize the following types of text:


  • Emails and email threads

  • Messages

  • Notifications and notification stacks

  • Notes

  • Posts

  • User-provided texts and news articles



Internally, Apple has tested multiple levels of summarization:


  • Topic-level summaries - consisting of three topic words, representing the key concepts of a text

  • 10 or 20-word summaries, which use clauses rather than full sentences

  • Paragraph-level summaries - concise three-sentence paragraph with no more than 60 words



Apple developed multiple summary types to account for specific use cases and UI elements, as certain summaries require more context and specific instructions to achieve the desired result.

More on the different summarization levels and text types



As explained in our exclusive report on Apple's on-device Ajax LLM, the large language model is capable of identifying keywords and sentences within a text. This means that it can identify units of language within a note, message, notification, or webpage, and create short summaries in the form of select keywords.

Colorful Siri icon, Safari and Messages app icons, and Spotlight Search bar on a dark background.
Apple's Ajax LLM is capable of performing topic-level summarizations entirely on-device



This simple selection of keywords is the basis for the software's topic-level summaries. For this level of summarization, Apple's AI is told to isolate three topic words, representing core concepts from the input text.

Although the Ajax LLM can handle topic-level summaries, advanced summarization, and more complex tasks appear to necessitate cloud-based processing. Apple may have recently partnered with OpenAI to facilitate advanced AI functionality, as indicated by a recent report.

Apart from topic-style summaries, Apple also experimented with 10 to 20-word summarization. With this type of summary, Apple's generative AI software is told to use clauses rather than complete sentences and to generate a result with no more than 10 or 20 words.

Paragraph-level summarization requires Apple's AI software to create a three-sentence concise paragraph consisting of no more than 60 words. Specific instructions are given based on the type of text the software is meant to summarize.

How does Apple instruct and guide its artificial intelligence?



Although Apple's approach to AI and LLMs does not include a chatbot, instructions used for training the software bear some resemblance to ChatGPT-style prompts. Apple has reportedly secured a licensing deal with OpenAI, meaning that we could see OpenAI's technology in Apple's new operating systems.

Bar graph showing major building projects by Roman emperors, displayed in a ChatGPT interface with various Roman Empire topics listed on the left sidebar.
Apple has created ChatGPT-style prompts for its AI



The company's instructions begin by directly stating that the AI is to assume the role of an expert in summarizing a specific type of text. The AI is told to maintain the role, or else its responses will not be considered useful or helpful - indicating that the overall goal is to assist the end users of Apple's operating systems.

When summarizing a stack of notifications, the AI has to focus on the common topic of all notifications within said stack. Similarly, when analyzing and creating a summary of user-supplied text such as an article, the software is instructed to focus on the main topic of the text in its entirety.

For texts provided by the end-user, the AI is instructed to refrain from using unnecessary prefaces and to exclude minor details. To generate a summary of recent messages or notification stacks, the software must put emphasis on key information - the names of the sender and recipient as well as the time and date.

In essence, this all means that Apple's virtual assistant - Siri - will be the main recipient of the company's LLM-powered summarization upgrades. Through generative AI and integration with core system applications, Siri will be able to do much more than ever before.

What else will Siri be able to do?



According to people familiar with the matter, Siri is also expected to receive significantly enhanced in-app functionality. This means that Apple's virtual assistant will be able to perform arguably complex tasks within various system applications.

A glowing, colorful, abstract orb with overlapping, translucent shapes on a dark background.
Siri will gain the ability to perform in-app tasks in apps such as Photos, Messages and Notes



In Photos, for instance, users will have the option to enhance individual photos, open specific types of images, play memories, and set their device wallpaper through Apple's virtual assistant.

Users could say something like "Enhance this photo" or "Show me photos from last year" to facilitate these actions. Siri will likely be able to display Photos from the user's library rather than immediately searching the web for images.

Similar Siri functionality is expected to become available within other system applications such as Messages and the built-in Notes app. The virtual assistant will be able to move individual notes, open and analyze documents, and send images or files via the Messages app.

The default Photos app will also receive some minor UI changes, as certain interface elements will be reorganized or renamed. The design change itself is little more than a stylistic choice, though, since other AI-powered editing features are the main update we can look forward to.

Generate and edit images with Apple's AI software



Apple's AI software is also capable of generating images in internal test environments. Now we have more details as to the different types of images the software can create.

Dog with polka dot bandana on a leash, standing on gravel near a fence, with blurry outdoor background and editing menu shown.
Apple's built-in Photos app will receive a new Clean Up tool, along with some UI changes



According to people familiar with the matter, the company's pre-release operating systems feature an entirely new image generation framework. With it, it's possible to create different types of images by employing artificial intelligence.

The new image generation framework references the following image types and styles:


  • Animation

  • Emoji

  • Illustration

  • Line Art

  • Sketch



As Apple has developed its own internal image generation app, along with a new framework for it - it's likely that the company will make this feature available to end users in some form. One report indicated that AI-generated emojis are in the works, and our findings corroborate this claim.

AI-powered image editing is also in the works, as Apple intends to replace its existing Retouch Tool with a new feature called Clean Up. The Clean Up option will allow users to remove objects from photos through the use of generative AI.

It's worth emphasizing, however, that the information comes from people familiar with pre-release versions of Apple's operating systems and test environments. Apple has been known to cancel, delay or rename new OS features ahead of release, so there's always a chance the features mentioned here won't become available to the general public.

Apple is set to debut its next-gen operating systems, such as iOS 18, iPadOS 18 and macOS 15, at its annual Worldwide Developers' Conference (WWDC) on June 10. At the event, we'll likely see a revamped Calculator along with upgraded versions of the built-in Calendar, Notes and Safari applications.




Read on AppleInsider

«1

Comments

  • Reply 1 of 21
    dutchlorddutchlord Posts: 263member
    Pff my expectations are zero as Siri cannot understand my language and cannot even perform simple tasks faultless. That’s why it is switched off. Besides, I don’t want any AI interpreting texts I receive or send on my device. I will switch it off when possible or I will stop using the device. 
    williamlondon
  • Reply 2 of 21
    eightzeroeightzero Posts: 3,134member
    What I'd *really* like for changes in apple mail is to make it easier and more reliable to simply set up. The host name, login, SSL, port stuff is a headache on any new installation, particularly on previous OS version. YMMV, but this AI system is not going to make my use of apple mail better.
    edited June 6 Oferwatto_cobra
  • Reply 3 of 21
    davebarnesdavebarnes Posts: 375member
    I don't care about AI features.
    I want 2 things:
    1. graybars
    2. the ability to change the subject of a received email (a là Eudora)
    williamlondon9secondkox2watto_cobra
  • Reply 4 of 21
    sunman42sunman42 Posts: 289member
    Allow me to express my skepticism that any of these changes will make my life better.
    williamlondon9secondkox2michelb76watto_cobra
  • Reply 5 of 21
    jas99jas99 Posts: 163member
    I think what we’re seeing with the AI “revolution“ is how different segments of the population deal with information and their lives.

    Apple users have typically been intelligent and capable people. Such people don’t want a machine managing their email – and making mistakes. Such people don’t need help summarizing large amount of information and understanding it. Such people don’t need help “making their email sound more professional “.

    Sadly, there are huge numbers of people who do need this sort of assistance. I think this is partly what’s driving the AI mania.

    I think Apple will implement some truly useful features using AI. Some of these AI features will be things that everybody, even traditional Apple customers, will find helpful. Also, Apple will be forced to include the fluffy, mass market AI features that the likes of Google and Microsoft make their main product.
    CuJoYYC9secondkox2watto_cobra
  • Reply 6 of 21
    The majority of the 'AI Frenzy' has everything to do with capitalism. First and foremost it gives Wall Street and companies a way to make it look and sound like they are deserving of more attention and a higher stock price. So mission accomplished. The market is dumb, and it overreacts and people, especially funds and investment banks, take a lot of cash out of this. The second thing is American Capitalism's love affair with not having employees so all profit can be cycled up to people who don't actually do work. And AI promises to relieve company of employing people so more money can be sucked from humans producing actual goods and services and put into the pockets of stock holders. So far AI has some efficiencies, in coding, and maybe research (Still GIGO). But mostly its hereto revitalize stocks. 
    gatorguyOfer9secondkox2watto_cobra
  • Reply 7 of 21
    emoelleremoeller Posts: 584member
    I agree with others - NONE OF THIS IS OF ANY VALUE until Siri actually captures and works with my voice, which it does not do now.   There needs to be some sort of training method to get Siri at six sigma recognition before any of this AI stuff will provide any meaningful value to users.

    Today simple transcription is barely at 60% accurate for me.   I have to scream at my HomePod in the morning to get it to play the Economist Intelligence podcast, even though I ask Siri to do this every single day right after I say "Hey Sire, what is my update?"  

    As I noted in the WSJ, during Apple's last earnings report the live screen transcription was a mess mis-stating what Tim had just stated.  Embarrassing to say the least.
    williamlondon
  • Reply 8 of 21
    OferOfer Posts: 265unconfirmed, member
    The majority of the 'AI Frenzy' has everything to do with capitalism. First and foremost it gives Wall Street and companies a way to make it look and sound like they are deserving of more attention and a higher stock price. So mission accomplished. The market is dumb, and it overreacts and people, especially funds and investment banks, take a lot of cash out of this. The second thing is American Capitalism's love affair with not having employees so all profit can be cycled up to people who don't actually do work. And AI promises to relieve company of employing people so more money can be sucked from humans producing actual goods and services and put into the pockets of stock holders. So far AI has some efficiencies, in coding, and maybe research (Still GIGO). But mostly its hereto revitalize stocks. 
    Agreed on all of the above! AI has the potential to automate boring manual things and freeing up all of us to delve more into creative endeavors. Instead it’s taking away from artists and creatives and leaving the boring manual stuff to humans. This isn’t how this should be going.
    watto_cobra
  • Reply 9 of 21
    cr41g_cr41g_ Posts: 1member
    How about fixing the completely borked  Mail rules message colorization of messages in folders before getting all "fancy"? The ability to categorize/prioritize messaging with color after the message has been binned with a rule would do way more for my productivity than any "interpretations/summarizations" AI might hork up. And I suspect would require significantly less FTE/hours (i.e., money), though it's probably not the "cool thing" to work on.

    And fwiw, I use flags for completely different things.
    williamlondonwatto_cobra
  • Reply 10 of 21
    danoxdanox Posts: 3,284member
    dutchlord said:
    Pff my expectations are zero as Siri cannot understand my language and cannot even perform simple tasks faultless. That’s why it is switched off. Besides, I don’t want any AI interpreting texts I receive or send on my device. I will switch it off when possible or I will stop using the device. 
    The English speaking countries are very lacking when a different language is required sadly but soon by the end of the decade AI will eliminate that problem.
    williamlondonwatto_cobra
  • Reply 11 of 21
    dewmedewme Posts: 5,678member
    It will be interesting to see how the AI responds to junk mail and spam. Maybe Apple will provide a check box to enable “Smart Ass Replies” instead of “Smart Replies.” In all seriousness, if AI based junk and space filtering can reduce unwanted email very significantly it will be a big win. 

    Hopefully Apple will provide switches for users to turn off any and all AI “features.”
    edited June 6 CuJoYYCwatto_cobra
  • Reply 12 of 21
    9secondkox29secondkox2 Posts: 3,030member
    Cool

    maybe Johnny depp can be the new “Clippy!”
    watto_cobra
  • Reply 13 of 21
    CheeseFreezeCheeseFreeze Posts: 1,331member
    I don’t email that much anymore. All my comms go through Slack and GSuite, Teams.

    Email is now primary for;
    - ordering and delivery confirmations
    - status updates 
    - (secondary authentication) login codes 
    - email lists 

    Actually writing mails? A couple every week. 

    So… perhaps Apple’s AI approach should be about automatically categorizing these types of emails.
    williamlondonwatto_cobra
  • Reply 14 of 21
    hexclockhexclock Posts: 1,305member
    I guess if you write emails for a living your job might get easier. 
     As long as the AI can’t lay flagstone patios, or rig lighting grids and audio stacks, I’m good. 
    gilly339secondkox2watto_cobra
  • Reply 15 of 21
    klink172klink172 Posts: 16member
    What about spam?!?
    9secondkox2watto_cobra
  • Reply 16 of 21
    gilly33gilly33 Posts: 444member
    Will Siri really get smart or are you just teasing us in this article?
    williamlondonwatto_cobra
  • Reply 17 of 21
    9secondkox29secondkox2 Posts: 3,030member
    The majority of the 'AI Frenzy' has everything to do with capitalism. First and foremost it gives Wall Street and companies a way to make it look and sound like they are deserving of more attention and a higher stock price. So mission accomplished. The market is dumb, and it overreacts and people, especially funds and investment banks, take a lot of cash out of this. The second thing is American Capitalism's love affair with not having employees so all profit can be cycled up to people who don't actually do work. And AI promises to relieve company of employing people so more money can be sucked from humans producing actual goods and services and put into the pockets of stock holders. So far AI has some efficiencies, in coding, and maybe research (Still GIGO). But mostly its hereto revitalize stocks. 
    Capitalism is great. The most successful way of doing things in history. 

    But… every now and then a product/service comes along that’s lame or short-lives, yet hyped. The market is people. And people are hard to predict. Sometimes people are wise. Sometimes foolish. But at least they’re free to be either rather than stuck in a system in which they are dictated to be force of government or by force of mob. 


    jas99TheSparklewatto_cobrabeowulfschmidt
  • Reply 18 of 21
    debonbondebonbon Posts: 11member
    I feel like this whole ai push by Apple is going to blow up in their face. I just hope to God I can turn it all off.
    9secondkox2williamlondonwatto_cobra
  • Reply 19 of 21
    fastasleepfastasleep Posts: 6,452member
    I want an AI feature that labels all the stupid comments in this forum as they are, because these features have nothing to do with voice recognition or current Siri capabilities in any way. I was literally just clearing emails and hoping that next week we get some intelligent message sorting, which from this article it sounds like we are. That's a good thing. Everyone here saying they'll just turn it off or complaining about it in advance are fools. 
    watto_cobra
  • Reply 20 of 21
    michelb76michelb76 Posts: 682member
    All of the email AI features that have been in Gmail for quite some time that nobody uses. Hope there will be more inspiring stuff coming out in the future. I guess it also just shows how limited in usefulness AI features still are.
    watto_cobra
Sign In or Register to comment.