Apple invention details deep Siri integration with Messages
Apple in a patent application published Thursday details a method of integrating an intelligent virtual assistant -- Siri -- into a messaging environment, allowing users to participate in text-based exchanges with the AI, while opening up automation and helper tools to traditional Messages conversations with friends.
Source: USPTO
As published by the U.S. Patent and Trademark Office, Apple's patent application for an "Intelligent automated assistant in a messaging environment" describes a method of meshing Siri with a graphical user interface designed for text messaging. Importantly, the integration would grant Siri the ability to not only converse with a single user, but also act as a third party in group chats.
The proposed system appears much more advanced than current Siri implementations that rely largely on speech recognition. As detailed in today's document, Siri integration would enable users to "talk" to the virtual assistant using any form of communication supported by Messages, which at this time includes text, audio, images and video.
Currently, Siri accepts voice commands via device microphones in a dedicated Siri UI. While the assistant can parse natural language input with a high degree of accuracy, mistakes do occur. In such cases, users can modify their query by tapping or selecting confirmation text and typing in the correct information. Siri also lacks the ability to process images in its current form.
As noted in the patent application, moving Siri to a messaging environment fosters all-new interactions. For example, a text-based UI is preferable in noisy environments or in quiet locations where audio interactions are not desired, like a library or movie theater.
Since Messages, and other messaging apps, present information in a chronological format, users would be able to review previous Siri interactions as traditional text histories. Additionally, Siri can reference available chat histories for contextual clues as it attempts to answer user queries. Currently, the Siri UI deletes a query thread once a session is complete.
Like existing implementations, contextual information can go beyond conversation history. For example, device location can be leveraged to answer a question like, "How did the Giants do?" A generated task flow first detects user intent, which in this case is an inquiry into a sports score. Depending on the location of the device, time and team schedule, Siri might return a score for the San Francisco Giants or New York Giants.
Subsequent messages can include links to internet websites relating to the sports score, or internal links to other apps.
Interestingly, the document mentions image and video recognition as viable use case scenarios for a Messages-based Siri. The patent application cites an image of a Volkswagen Beetle, which might be "sent" to Siri by a user as part of a query. In one example, a user asks Siri to gather information on pricing, prompting the virtual assistant to break down the image using identifying characteristics, search the web for current prices and return an answer in the form of a text.
Other examples of image-related queries include "Where is this?" "What insect is this?" "Which company uses this logo?" and more. In some cases, video can be processed by recognizing sounds within a given clip, which is the same method mentioned for parsing audio queries.
With image and audio recognition capabilities enabled, Siri can also "remember" user preferences. For example, if a user sends a picture of a bottle of wine and says, "I like this wine," Siri will make a note for later reference. In the same vein, users can enter blocks of text for storage, such as contact information, which the virtual assistant can file away for later retrieval.
Perhaps most intriguing is multi-user conversations in which Siri plays a role as an active participant. Inviting Siri to join a chat, as one would a third person in Messages, allows the AI to offer its services to both human users. For example, a first user might ask about nearby Chinese restaurants, a query to which Siri would respond with a list of appropriate venues. A second person might want to narrow the list to cheaper eateries and, once a particular restaurant is decided, create a calendar entry for storage on both user devices.
Going further, Siri can later remind each person of the upcoming meeting and suggest transportation options. In examples offered in today's document, each Siri assistant has its own identity. For instance, when reminding a user of a calendar invite, Siri might send a message to a first user saying, "Don't forget about dinner at 7 p.m.," while sending another message to a second user saying, "John Smith's Siri reminds you about dinner tonight at 7 p.m."
In theory, Siri could become an integral third party for Messages chats.
Apple's plans for the invention are unclear at this time. An unverified rumor surfaced in March suggesting Messages in the next-generation "iOS 11" operating system will support some form of Siri integration, though the source report read like speculation.
Thought to be based on recent patent filings, the supposed feature will allow Siri to eavesdrop on Messages conversations in order to provide a device owner with useful information. For example, when a user discusses eating a meal with friends, Siri might pop in and suggest nearby restaurants, ask to schedule reservations or arrange for an Uber. Such capabilities are mentioned in today's patent filing.
Integrating AI assistants into messaging utilities is not unprecedented. Google last year launched the Allo messaging app with Google Assistant integration, granting users in-app access to what boils down to an advanced chat bot. Similarly, popular workplace messaging platform Slack features its own -- less advanced -- chat bot called Slackbot, which can answer common questions and perform rudimentary tasks.
Apple's patent application detailing Siri integration for Messages was first filed for in May 2016 and credits Petr Karashchuk, Tomas A. Vega Galvez and Thomas R. Gruber as its inventors.
Source: USPTO
As published by the U.S. Patent and Trademark Office, Apple's patent application for an "Intelligent automated assistant in a messaging environment" describes a method of meshing Siri with a graphical user interface designed for text messaging. Importantly, the integration would grant Siri the ability to not only converse with a single user, but also act as a third party in group chats.
The proposed system appears much more advanced than current Siri implementations that rely largely on speech recognition. As detailed in today's document, Siri integration would enable users to "talk" to the virtual assistant using any form of communication supported by Messages, which at this time includes text, audio, images and video.
Currently, Siri accepts voice commands via device microphones in a dedicated Siri UI. While the assistant can parse natural language input with a high degree of accuracy, mistakes do occur. In such cases, users can modify their query by tapping or selecting confirmation text and typing in the correct information. Siri also lacks the ability to process images in its current form.
As noted in the patent application, moving Siri to a messaging environment fosters all-new interactions. For example, a text-based UI is preferable in noisy environments or in quiet locations where audio interactions are not desired, like a library or movie theater.
Since Messages, and other messaging apps, present information in a chronological format, users would be able to review previous Siri interactions as traditional text histories. Additionally, Siri can reference available chat histories for contextual clues as it attempts to answer user queries. Currently, the Siri UI deletes a query thread once a session is complete.
Like existing implementations, contextual information can go beyond conversation history. For example, device location can be leveraged to answer a question like, "How did the Giants do?" A generated task flow first detects user intent, which in this case is an inquiry into a sports score. Depending on the location of the device, time and team schedule, Siri might return a score for the San Francisco Giants or New York Giants.
Subsequent messages can include links to internet websites relating to the sports score, or internal links to other apps.
Interestingly, the document mentions image and video recognition as viable use case scenarios for a Messages-based Siri. The patent application cites an image of a Volkswagen Beetle, which might be "sent" to Siri by a user as part of a query. In one example, a user asks Siri to gather information on pricing, prompting the virtual assistant to break down the image using identifying characteristics, search the web for current prices and return an answer in the form of a text.
Other examples of image-related queries include "Where is this?" "What insect is this?" "Which company uses this logo?" and more. In some cases, video can be processed by recognizing sounds within a given clip, which is the same method mentioned for parsing audio queries.
With image and audio recognition capabilities enabled, Siri can also "remember" user preferences. For example, if a user sends a picture of a bottle of wine and says, "I like this wine," Siri will make a note for later reference. In the same vein, users can enter blocks of text for storage, such as contact information, which the virtual assistant can file away for later retrieval.
Perhaps most intriguing is multi-user conversations in which Siri plays a role as an active participant. Inviting Siri to join a chat, as one would a third person in Messages, allows the AI to offer its services to both human users. For example, a first user might ask about nearby Chinese restaurants, a query to which Siri would respond with a list of appropriate venues. A second person might want to narrow the list to cheaper eateries and, once a particular restaurant is decided, create a calendar entry for storage on both user devices.
Going further, Siri can later remind each person of the upcoming meeting and suggest transportation options. In examples offered in today's document, each Siri assistant has its own identity. For instance, when reminding a user of a calendar invite, Siri might send a message to a first user saying, "Don't forget about dinner at 7 p.m.," while sending another message to a second user saying, "John Smith's Siri reminds you about dinner tonight at 7 p.m."
In theory, Siri could become an integral third party for Messages chats.
Apple's plans for the invention are unclear at this time. An unverified rumor surfaced in March suggesting Messages in the next-generation "iOS 11" operating system will support some form of Siri integration, though the source report read like speculation.
Thought to be based on recent patent filings, the supposed feature will allow Siri to eavesdrop on Messages conversations in order to provide a device owner with useful information. For example, when a user discusses eating a meal with friends, Siri might pop in and suggest nearby restaurants, ask to schedule reservations or arrange for an Uber. Such capabilities are mentioned in today's patent filing.
Integrating AI assistants into messaging utilities is not unprecedented. Google last year launched the Allo messaging app with Google Assistant integration, granting users in-app access to what boils down to an advanced chat bot. Similarly, popular workplace messaging platform Slack features its own -- less advanced -- chat bot called Slackbot, which can answer common questions and perform rudimentary tasks.
Apple's patent application detailing Siri integration for Messages was first filed for in May 2016 and credits Petr Karashchuk, Tomas A. Vega Galvez and Thomas R. Gruber as its inventors.
Comments
seriously the novelty was fun when I got my 4s. I honestly think it's gotten worse, BUT Apple thinks Siri should control more and more products like HomeKit. I would truly just be happy if Siri understood a clearly annunciated command or sentence...I can't adequately express how much contempt I have for the tech company I used to love that the price keeps going up but the justification is really stupid gimmicks. I really really don't even want to hear Siri do a beat box! Don't care doesn't help anything!
as for Siri if you are even slightly serious go look in the mirror and slap yourself until you stop acting like Apple is the best. It used to be. I got the first gen MacBookpro and jobs was right "it just works" - very little in Apple lineup does that or comes close. But the price has blown through the roof!
among the things that barely works at all is Siri - and if you can't see that you don't know what you're talking about. I call Apple my back up religion, but good lord the adulation of crappier and crappier gimmicks pisses me off. Just stop
I'm not about to respond to your BS so sorry. Buy what you like...its your money.
Nice find! Potential of Siri only starting to be appreciated by the general crowd.
It could be THE centerpiece for Apple’s growth in services. In my view, AAPL has more to gain and GOOGL and AMZN more to loose. A more capable Siri assistant is a threat to the walled gardens of Google’s search and Amazon’s e-commerce. Might be worthy of an AI series? How AAPL might into grow it.
Sidenote: Shopping on Amazon becoming more like visiting a casino. Some may like that. Not me. I wasted half an hour to no avail trying to find an old fashioned natural soap (after Alleppo soaps and Marseilles soap I previously got there weren’t properly labeled).
My dream in this case is next-gen Siri intell asst takes me to say the best bar of soap in my price range. And is daily on lookout for deals on things I like (without me asking and knowing my tolerance thresholds). Maybe Apple can start by forming alliances with reliable stores in top margin areas: like health and beauty.
What vendor wouldn’t want to tap into AAPL’s base? The App Store is just a start, twenty years from now a fraction…
Various experts need to hash it out. But it seems Apple holds many keys nearing 2020: a billion premium consumers. Privacy and security a greater issue as next-gen digital assistants become part of your life 24/7.
P.S. I’m not saying Apple mimic the warehouses of Amazon (as AMZN is opening its own stores a la AAPL) But AAPL can cherrypick lucrative e commerce areas and let others deal with fulfillment.
Siri was very competitive, ahead of Cortana, but behind Googl in a few instances. I didn't see actual report, just short summary.