Apple invention details deep Siri integration with Messages

Posted:
in General Discussion edited May 2017
Apple in a patent application published Thursday details a method of integrating an intelligent virtual assistant -- Siri -- into a messaging environment, allowing users to participate in text-based exchanges with the AI, while opening up automation and helper tools to traditional Messages conversations with friends.


Source: USPTO


As published by the U.S. Patent and Trademark Office, Apple's patent application for an "Intelligent automated assistant in a messaging environment" describes a method of meshing Siri with a graphical user interface designed for text messaging. Importantly, the integration would grant Siri the ability to not only converse with a single user, but also act as a third party in group chats.

The proposed system appears much more advanced than current Siri implementations that rely largely on speech recognition. As detailed in today's document, Siri integration would enable users to "talk" to the virtual assistant using any form of communication supported by Messages, which at this time includes text, audio, images and video.

Currently, Siri accepts voice commands via device microphones in a dedicated Siri UI. While the assistant can parse natural language input with a high degree of accuracy, mistakes do occur. In such cases, users can modify their query by tapping or selecting confirmation text and typing in the correct information. Siri also lacks the ability to process images in its current form.

As noted in the patent application, moving Siri to a messaging environment fosters all-new interactions. For example, a text-based UI is preferable in noisy environments or in quiet locations where audio interactions are not desired, like a library or movie theater.

Since Messages, and other messaging apps, present information in a chronological format, users would be able to review previous Siri interactions as traditional text histories. Additionally, Siri can reference available chat histories for contextual clues as it attempts to answer user queries. Currently, the Siri UI deletes a query thread once a session is complete.

Like existing implementations, contextual information can go beyond conversation history. For example, device location can be leveraged to answer a question like, "How did the Giants do?" A generated task flow first detects user intent, which in this case is an inquiry into a sports score. Depending on the location of the device, time and team schedule, Siri might return a score for the San Francisco Giants or New York Giants.

Subsequent messages can include links to internet websites relating to the sports score, or internal links to other apps.




Interestingly, the document mentions image and video recognition as viable use case scenarios for a Messages-based Siri. The patent application cites an image of a Volkswagen Beetle, which might be "sent" to Siri by a user as part of a query. In one example, a user asks Siri to gather information on pricing, prompting the virtual assistant to break down the image using identifying characteristics, search the web for current prices and return an answer in the form of a text.

Other examples of image-related queries include "Where is this?" "What insect is this?" "Which company uses this logo?" and more. In some cases, video can be processed by recognizing sounds within a given clip, which is the same method mentioned for parsing audio queries.

With image and audio recognition capabilities enabled, Siri can also "remember" user preferences. For example, if a user sends a picture of a bottle of wine and says, "I like this wine," Siri will make a note for later reference. In the same vein, users can enter blocks of text for storage, such as contact information, which the virtual assistant can file away for later retrieval.




Perhaps most intriguing is multi-user conversations in which Siri plays a role as an active participant. Inviting Siri to join a chat, as one would a third person in Messages, allows the AI to offer its services to both human users. For example, a first user might ask about nearby Chinese restaurants, a query to which Siri would respond with a list of appropriate venues. A second person might want to narrow the list to cheaper eateries and, once a particular restaurant is decided, create a calendar entry for storage on both user devices.




Going further, Siri can later remind each person of the upcoming meeting and suggest transportation options. In examples offered in today's document, each Siri assistant has its own identity. For instance, when reminding a user of a calendar invite, Siri might send a message to a first user saying, "Don't forget about dinner at 7 p.m.," while sending another message to a second user saying, "John Smith's Siri reminds you about dinner tonight at 7 p.m."

In theory, Siri could become an integral third party for Messages chats.





Apple's plans for the invention are unclear at this time. An unverified rumor surfaced in March suggesting Messages in the next-generation "iOS 11" operating system will support some form of Siri integration, though the source report read like speculation.

Thought to be based on recent patent filings, the supposed feature will allow Siri to eavesdrop on Messages conversations in order to provide a device owner with useful information. For example, when a user discusses eating a meal with friends, Siri might pop in and suggest nearby restaurants, ask to schedule reservations or arrange for an Uber. Such capabilities are mentioned in today's patent filing.

Integrating AI assistants into messaging utilities is not unprecedented. Google last year launched the Allo messaging app with Google Assistant integration, granting users in-app access to what boils down to an advanced chat bot. Similarly, popular workplace messaging platform Slack features its own -- less advanced -- chat bot called Slackbot, which can answer common questions and perform rudimentary tasks.

Apple's patent application detailing Siri integration for Messages was first filed for in May 2016 and credits Petr Karashchuk, Tomas A. Vega Galvez and Thomas R. Gruber as its inventors.
«1

Comments

  • Reply 1 of 26
    DracarysDracarys Posts: 72member
    So for all intents and purposes what Google does with Google Assistant in Allo...
    db22559anantksundaram
  • Reply 2 of 26
    Sure why not.
  • Reply 3 of 26
    calicali Posts: 3,494member
    Been wanting Siri to do this for a while. The "what am I looking at?" Part. 
  • Reply 4 of 26
    macxpressmacxpress Posts: 5,801member
    I hope they did more than just this with Siri...it needs help with a lot of things, not just one. If Apple really wants to improve upon your operating systems, make Siri your #1 priority. The damn phone does everything else you'd ever want it to anyways. 
    edited May 2017 argonaut
  • Reply 5 of 26
    rfrmacrfrmac Posts: 89member
    Not sure when we'll see it.  I'm just glad something is being done to increase Siri's capabilities.  I'm just glad to see anything new from Apple.
    db22559
  • Reply 6 of 26
    irelandireland Posts: 17,798member
    All this sounds great, but I'd like improved voice recognition for this part of the world first.
    db22559
  • Reply 7 of 26
    irelandireland Posts: 17,798member
    macxpress said:
    I hope they did more than just this with Siri...it needs help with a lot of things, not just one. If Apple really wants to improve upon your operating systems, make Siri you're #1 priority. The damn phone does everything else you'd ever want it to anyways. 
    Ever?
    db22559
  • Reply 8 of 26
    SpamSandwichSpamSandwich Posts: 33,407member
    Siri is so incapable these days, it's difficult to imagine ever using the capabilities described in this patent. I'd love to be proven wrong.
    db22559anantksundaram
  • Reply 9 of 26
    macxpressmacxpress Posts: 5,801member
    ireland said:
    macxpress said:
    I hope they did more than just this with Siri...it needs help with a lot of things, not just one. If Apple really wants to improve upon your operating systems, make Siri you're #1 priority. The damn phone does everything else you'd ever want it to anyways. 
    Ever?
    What do you want next...for it to make breakfast or something for you? The point is...the phone is very capable and all we really see year after year is faster CPU/GPU, and a better camera. What else is there to do with the hardware other than play with different designs so it looks different? The hardware shouldn't be the focus anymore...its the software that people use all the time. Focus on that. 
    edited May 2017
  • Reply 10 of 26
    db22559db22559 Posts: 3unconfirmed, member
    So, what angers me about all Siri news I've heard is that it's either improving Siri's tonal responses (don't care...at all) or integration with native apps (sure why not make Siri slightly useful). Thing is I have no southern accent or really any speech quirks that should cause confusion for Siri, but there is only one command she understands from me "Siri set a timer for.." for example, asking for directions to work while driving? 70% chance she will tell me there's no contact for "work" so I cuss her ineptitude and search for directions in traffic. Not only twice as time consuming as doing it myself but probably more dangerous too. Another one is "hey Siri add haircut appointment tomorrow at 9 am" response: "Here's your 9 am haircut with your mono" (added to today's schedule). So instead of being even slightly useful or even just useless, Siri is negative usefulness because after she gets it wrong you have to go to iCal, delete her stuuuuupid mistake, then enter it manually on the correct day. I am so SO sick of Apple's self praise fetish over crap that just doesn't work. Siri, aside from the sassy quips responding to what does the fox say, Siri SUUUUUUCKS. Apple has limited Cortana a lot but none of the other ai apps have these issues that consistently misunderstand what I say and what should be obvious. Perhaps it's because they understand the key term in AI is intelligence. So far Siri only qualifies as artificial.

    seriously the novelty was fun when I got my 4s. I honestly think it's gotten worse, BUT Apple thinks Siri should control more and more products like HomeKit. I would truly just be happy if Siri understood a clearly annunciated command or sentence...I can't adequately express how much contempt I have for the tech company I used to love that the price keeps going up but the justification is really stupid gimmicks. I really really don't even want to hear Siri do a beat box! Don't care doesn't help anything!
    profssrfinkanantksundaram
  • Reply 11 of 26
    db22559db22559 Posts: 3unconfirmed, member
    Dracarys said:
    So for all intents and purposes what Google does with Google Assistant in Allo...
    YES! I'm fine if Apple perfects existing tech but the quality comparison is not even close anymore. 
  • Reply 12 of 26
    db22559db22559 Posts: 3unconfirmed, member
    macxpress said:
    ireland said:
    macxpress said:
    I hope they did more than just this with Siri...it needs help with a lot of things, not just one. If Apple really wants to improve upon your operating systems, make Siri you're #1 priority. The damn phone does everything else you'd ever want it to anyways. 
    Ever?
    What do you want next...for it to make breakfast or something for you? The point is...the phone is very capable and all we really see year after year is faster CPU/GPU, and a better camera. What else is there to do with the hardware other than play with different designs so it looks different? The hardware shouldn't be the focus anymore...its the software that people use all the time. Focus on that. 
    Excuse me?! Shouldn't be focus anymore? Let me enlighten you - it hasn't been in a very long time. Examples? MacBook Pro is now about 1k more than the model I currently own (last one released). The specs however are less powerful than the one I'm using!!! ARE YOU KIDDING ME?! How about the watch tethered to iPhone so you have to buy both. Smart money maker, but I've been waiting to get a watch until they include lte chip. When did other companies do that? two Apple Watches ago! On top of that the Samsung gear looks gorgeous next to Apple Watch! Price difference? Last year nearly half Apple. How about the pencil? $100 whew for that price can't wait to use it and see how awesome it is. I assumed it connected seamlessly like the AirPods (another overpriced p.o.s.) but here's the thing, it disconnects constantly, and they even have a disclaimer about it! Moreover it can only be reconnected by removing the cap, pause while you chase the cap that just rolled away, then inserting the pencil into the lightning port butt first. Seriously?! Whoa talk about cutting edge!

    as for Siri if you are even slightly serious go look in the mirror and slap yourself until you stop acting like Apple is the best. It used to be. I got the first gen MacBookpro and jobs was right "it just works" - very little in Apple lineup does that or comes close. But the price has blown through the roof!

    among the things that barely works at all is Siri - and if you can't see that you don't know what you're talking about. I call Apple my back up religion, but good lord the adulation of crappier and crappier gimmicks pisses me off. Just stop 
    anantksundaramavon b7
  • Reply 13 of 26
    macxpressmacxpress Posts: 5,801member
    db22559 said:
    macxpress said:
    ireland said:
    macxpress said:
    I hope they did more than just this with Siri...it needs help with a lot of things, not just one. If Apple really wants to improve upon your operating systems, make Siri you're #1 priority. The damn phone does everything else you'd ever want it to anyways. 
    Ever?
    What do you want next...for it to make breakfast or something for you? The point is...the phone is very capable and all we really see year after year is faster CPU/GPU, and a better camera. What else is there to do with the hardware other than play with different designs so it looks different? The hardware shouldn't be the focus anymore...its the software that people use all the time. Focus on that. 
    Excuse me?! Shouldn't be focus anymore? Let me enlighten you - it hasn't been in a very long time. Examples? MacBook Pro is now about 1k more than the model I currently own (last one released). The specs however are less powerful than the one I'm using!!! ARE YOU KIDDING ME?! How about the watch tethered to iPhone so you have to buy both. Smart money maker, but I've been waiting to get a watch until they include lte chip. When did other companies do that? two Apple Watches ago! On top of that the Samsung gear looks gorgeous next to Apple Watch! Price difference? Last year nearly half Apple. How about the pencil? $100 whew for that price can't wait to use it and see how awesome it is. I assumed it connected seamlessly like the AirPods (another overpriced p.o.s.) but here's the thing, it disconnects constantly, and they even have a disclaimer about it! Moreover it can only be reconnected by removing the cap, pause while you chase the cap that just rolled away, then inserting the pencil into the lightning port butt first. Seriously?! Whoa talk about cutting edge!

    as for Siri if you are even slightly serious go look in the mirror and slap yourself until you stop acting like Apple is the best. It used to be. I got the first gen MacBookpro and jobs was right "it just works" - very little in Apple lineup does that or comes close. But the price has blown through the roof!

    among the things that barely works at all is Siri - and if you can't see that you don't know what you're talking about. I call Apple my back up religion, but good lord the adulation of crappier and crappier gimmicks pisses me off. Just stop 
    Welcome to the forum! Its nice to see that people still sign up to troll. This isn't MacRumors...go over there. 

    I'm not about to respond to your BS so sorry. Buy what you like...its your money. 
    edited May 2017 StrangeDayswatto_cobra
  • Reply 14 of 26
    FolioFolio Posts: 698member

    Nice find! Potential of Siri only starting to be appreciated by the general crowd. 

    It could be THE centerpiece for Apple’s growth in services. In my view, AAPL has more to gain and GOOGL and AMZN more to loose. A more capable Siri assistant is a threat to the walled gardens of Google’s search and Amazon’s e-commerce. Might be worthy of an AI series? How AAPL might into grow it. 


    Sidenote: Shopping on Amazon becoming more like visiting a casino. Some may like that. Not me. I wasted half an hour to no avail trying to find an old fashioned natural soap (after Alleppo soaps and Marseilles soap I previously got there weren’t properly labeled). 

    My dream in this case is next-gen Siri intell asst takes me to say the best bar of soap in my price range. And is daily on lookout for deals on things I like (without me asking and knowing my tolerance thresholds). Maybe Apple can start by forming alliances with reliable stores in top margin areas: like health and beauty. 

    What vendor wouldn’t want to tap into AAPL’s base? The App Store is just a start, twenty years from now a fraction…

    Various experts need to hash it out. But it seems Apple holds many keys nearing 2020: a billion premium consumers. Privacy and security a greater issue as next-gen digital assistants become part of your life 24/7.


    P.S. I’m not saying Apple mimic the warehouses of Amazon (as AMZN is opening its own stores a la AAPL) But AAPL can cherrypick lucrative e commerce areas and let others deal with fulfillment. 

  • Reply 15 of 26
    FolioFolio Posts: 698member
    Guess it depends on your interactions. To me Siri seems to be getting more capable and natural. For instance, adding a bit of personality to sports scores, like the Washington Wizards are "getting crushed". There was news report a week or two ago of Google Asst. Siri Cortana. I believe five categories of search. 

    Siri was very competitive, ahead of Cortana, but behind Googl in a few instances. I didn't see actual report, just short summary.
    watto_cobra
  • Reply 16 of 26
    Rayz2016Rayz2016 Posts: 6,957member
    db22559 said:
    macxpress said:
    ireland said:
    macxpress said:
    I hope they did more than just this with Siri...it needs help with a lot of things, not just one. If Apple really wants to improve upon your operating systems, make Siri you're #1 priority. The damn phone does everything else you'd ever want it to anyways. 
    Ever?
    What do you want next...for it to make breakfast or something for you? The point is...the phone is very capable and all we really see year after year is faster CPU/GPU, and a better camera. What else is there to do with the hardware other than play with different designs so it looks different? The hardware shouldn't be the focus anymore...its the software that people use all the time. Focus on that. 
    Excuse me?! Shouldn't be focus anymore? Let me enlighten you - it hasn't been in a very long time. Examples? MacBook Pro is now about 1k more than the model I currently own (last one released). The specs however are less powerful than the one I'm using!!! ARE YOU KIDDING ME?! How about the watch tethered to iPhone so you have to buy both. Smart money maker, but I've been waiting to get a watch until they include lte chip. When did other companies do that? two Apple Watches ago! On top of that the Samsung gear looks gorgeous next to Apple Watch! Price difference? Last year nearly half Apple. How about the pencil? $100 whew for that price can't wait to use it and see how awesome it is. I assumed it connected seamlessly like the AirPods (another overpriced p.o.s.) but here's the thing, it disconnects constantly, and they even have a disclaimer about it! Moreover it can only be reconnected by removing the cap, pause while you chase the cap that just rolled away, then inserting the pencil into the lightning port butt first. Seriously?! Whoa talk about cutting edge!

    as for Siri if you are even slightly serious go look in the mirror and slap yourself until you stop acting like Apple is the best. It used to be. I got the first gen MacBookpro and jobs was right "it just works" - very little in Apple lineup does that or comes close. But the price has blown through the roof!

    among the things that barely works at all is Siri - and if you can't see that you don't know what you're talking about. I call Apple my back up religion, but good lord the adulation of crappier and crappier gimmicks pisses me off. Just stop 
    And again. 



    macxpressStrangeDayswatto_cobra
  • Reply 17 of 26
    macxpressmacxpress Posts: 5,801member
    I mean if the new MacBook Pro was that bad then how the hell did Apple sell more than any other MacBook Pro in history? That includes, the outgoing model. Its obvious that the majority think its priced well or else they wouldn't buy it. The majority think its well spec'd for their use or else they wouldn't buy it. Just because it has an Apple logo on it, doesn't mean people will pay any price for something. There have been products in the past that proved this. I'm sorry Apple doesn't engineer a personal Mac for everyone tailored to their needs. Apparently to some, that used to happen??? Maybe when Steve was alive? 
    edited May 2017 StrangeDayswatto_cobra
  • Reply 18 of 26
    StrangeDaysStrangeDays Posts: 12,834member
    db22559 said:
    macxpress said:
    ireland said:
    macxpress said:
    I hope they did more than just this with Siri...it needs help with a lot of things, not just one. If Apple really wants to improve upon your operating systems, make Siri you're #1 priority. The damn phone does everything else you'd ever want it to anyways. 
    Ever?
    What do you want next...for it to make breakfast or something for you? The point is...the phone is very capable and all we really see year after year is faster CPU/GPU, and a better camera. What else is there to do with the hardware other than play with different designs so it looks different? The hardware shouldn't be the focus anymore...its the software that people use all the time. Focus on that. 
    Excuse me?! Shouldn't be focus anymore? Let me enlighten you - it hasn't been in a very long time. Examples? MacBook Pro is now about 1k more than the model I currently own (last one released). The specs however are less powerful than the one I'm using!!! ARE YOU KIDDING ME?! How about the watch tethered to iPhone so you have to buy both. Smart money maker, but I've been waiting to get a watch until they include lte chip. When did other companies do that? two Apple Watches ago! On top of that the Samsung gear looks gorgeous next to Apple Watch! Price difference? Last year nearly half Apple. How about the pencil? $100 whew for that price can't wait to use it and see how awesome it is. I assumed it connected seamlessly like the AirPods (another overpriced p.o.s.) but here's the thing, it disconnects constantly, and they even have a disclaimer about it! Moreover it can only be reconnected by removing the cap, pause while you chase the cap that just rolled away, then inserting the pencil into the lightning port butt first. Seriously?! Whoa talk about cutting edge!

    as for Siri if you are even slightly serious go look in the mirror and slap yourself until you stop acting like Apple is the best. It used to be. I got the first gen MacBookpro and jobs was right "it just works" - very little in Apple lineup does that or comes close. But the price has blown through the roof!

    among the things that barely works at all is Siri - and if you can't see that you don't know what you're talking about. I call Apple my back up religion, but good lord the adulation of crappier and crappier gimmicks pisses me off. Just stop 
    Hello. Please pass the reefer.
    watto_cobra
  • Reply 19 of 26
    glowearthglowearth Posts: 11member
    This would be awesome for Deaf, Hard of Hearing and Speech Impaired to use Siri based text so they can communicate with Siri. I'm glad that Apple finally invent this. I've been wait for long time because at first Siri is worthless for Deaf, Hard of Hearing and Speech Impaired. 
    watto_cobra
  • Reply 20 of 26
    I echo the comments above. I want SIRI to work well because I can see it being mroe useful if it did. I have an Apple watch, 2 iphones, tv and ipad and macbooks/imac and unless Im yelling at siri it often times doesn't respond. "Hey Siri" is awkward because I don't begin sentences with the word Hey. Alexa is a more natural word and lets me have a conversation rather than saying something awkward and waiting for my UI to pop up. Getting tired if it really and its truly limiting my appreciation of the tech. While Alexa is limited its still more responsive and the UI interaction is better.
    edited May 2017 anantksundaram
Sign In or Register to comment.