Apple considering offline mode for Siri that could process voice locally on an iPhone

Posted:
in iOS
Apple wants Siri to become more useful to users when not connected to the internet, including the possibility of an offline mode that does not rely on a backend server to assist with voice recognition or performing the required task, one that would be entirely performed on the user's device.




The way Siri typically works is that it listens to the user command when prompted, sends anonymized received speech data to Apple's servers, first converting from audio to plain text, then interpreting the command and sending the result back to the user's iPhone or iPad. The process of performing speech recognition is intensive, and is offloaded from devices to Apple's servers as it isn't necessarily capable of being performed on an item like an iPhone.

In the case of Siri, the reliance on servers means the virtual assistant isn't available if there isn't some form of internet connection, though the "Hey Siri" prompt will still trigger the service, if only to then display the offline message. According to an Apple patent application filing published on Thursday for an "Offline personal assistant," things may change in the future.

Rather than connected to Apple's servers, the filing suggests the speech-to-text processing and validation could happen on the device itself. On hearing the user make a request, the device in question will be capable of determining the task via onboard natural language processing, working out if the requested task as it hears it is useful, then performing it.

Apple suggests the use of an onboard system of modules to handle digital assistant queries that does not connect to the outside world. The collection of modules includes elements for speech synthesis, dialog processing, phonetic alphabet conversion based on a default vocabulary and user-created data, and a natural language processing module, among other items.

Depending on the recognized words and the structure of the query, the offline digital assistant could then perform a number of predefined tasks stored in the task flow processing module or service processing module. These two stores contain models for commonly-requested tasks, such as setting a timer or to play a song saved to the device, with the appropriate task model performed depending on the request.

Apple patent application diagram showing modules within a digital assistant
Apple patent application diagram showing modules within a digital assistant


As part of the filing, Apple does suggest the use of machine learning mechanisms to perform specific types of task, including natural language processing. Such systems would be able to understand contextual information, making them extremely useful for determining what the user wants from a limited amount of on-device data, without accessing the internet.

Even with the on-device processing for offline use, the proposed system would still be able to go online if a connection is available, making the same system work in two different ways depending on the available connectivity.

Having access to local and remote processing would also provide the digital assistant two possible interpretations for the user's speech. In such cases, the system would determine a usefulness score for the locally-processed interpretation as well as that of the online-processed version, and perform the task based on which of the two processes scored higher.

This would effectively give the assistant a backup option, allowing the server-based processing to be used as an alternate interpretation if the locally-performed processing on the request comes up with an unusable result. In the event the server version times out or becomes unavailable, the local version is still available to use for the task.

Typically Apple files for many patents on a weekly basis, and while the concepts described do suggest areas where Apple has an interest, they are not guarantees that such systems will be included in the company's future products.

In the case of offline Siri, it seems quite plausible for Apple to go down this route. In its more recent iPhone launches, the A-series processor has included the Neural Engine, which is intended to perform computationally intensive tasks such as Face ID authentication and powering photography-related features, but it can also be used for tasks that use machine learning.

Considering the power of the second-generation Neural Engine used in the 2018 iPhone releases, it is entirely possible for an offline Siri to take advantage of the available power and machine learning capabilities to perform offline processing.

An offline mode is not the only way Apple could improve Siri, as it has worked on a number of different concepts to make the digital assistant better. One October patent explains how Siri could recognize specific users by their "voice print," with Apple also looking into enabling multi-user support with personalized responses.

Siri has already received some improvements relating to its recognition capabilities, including geographical voice models to help aid with regional queries and local language quirks. More visible to consumers is the introduction of Siri Shortcuts in iOS 12, which allows users to make their own personal Siri commands.
«13

Comments

  • Reply 1 of 42
    MikeLVMikeLV Posts: 3unconfirmed, member
    Finally! Siri goes into dummy mode as soon as my connection gets weak, which renders it useless for Carplay in many instances. Siri should not have to rely on an internet connection for the most basic commands.
    redgeminipataddtaddaaronsullivanseanjBigDann
  • Reply 2 of 42
    lkrupplkrupp Posts: 7,162member
    This looks like a solution in need of a problem to me. I would think the number of iOS devices, especially iPhones, NOT connected to the Internet is a minuscule minority indeed. My iOS devices are connected 24/7 via WiFi or LTE.
    hodarelijahg
  • Reply 3 of 42
    MacProMacPro Posts: 18,309member
    Maybe this is as much to do with dealing with Siri overloads as when no internet connection.  It would switch you over to local use if the servers are overtaxed as I suspect is happening more and more these days.  There must be many questions that can be redirected locally if trivial.  That would require some way of the local system knowing it is trivial though ...
    edited November 2018 BigDannjony0
  • Reply 4 of 42
    hodarhodar Posts: 272member
    So, if you think Siri is stupid now, just wait.

    Try asking Siri a very basic question, like "How old am I", or "When will I be 59 1/2 years old".  Siri will suggest websites; which is asinine.  Siri knows my birthday, anniversary, the birthday of my kids, friends and neighbors (as long as I have the information in my contacts); but is unable to use that date, and a calendar to answer very basic questions.  Voice recognition on Siri has improved - but Siri's ability to do anything useful with that information is almost 5 years behind both Google, and Amazon - which is pathetic - since Apple practically invented a useful Virtual Assistant.

    Now, between Cortana, Echo and Google Echo - Siri is pretty much a dead last competitor; when it should be the BEST - by far.
    elijahgpatchythepirateFatmanairnerd
  • Reply 5 of 42
    ne1ne1 Posts: 37member
    lkrupp said:
    This looks like a solution in need of a problem to me. I would think the number of iOS devices, especially iPhones, NOT connected to the Internet is a minuscule minority indeed. My iOS devices are connected 24/7 via WiFi or LTE.
    Not really. You’re basing this on your own experience but despite living in the populated Northeast corridor, there are inevitably weak cell phone signals, connection interruptions at rush hour, WiFi outages, etc. It’s about time Siri usage went offline. 
    mike1napoleon_phoneapartaaronsullivanwatto_cobraBigDannjony0
  • Reply 6 of 42
    lkrupp said:
    This looks like a solution in need of a problem to me. I would think the number of iOS devices, especially iPhones, NOT connected to the Internet is a minuscule minority indeed. My iOS devices are connected 24/7 via WiFi or LTE.
    Well, yes, but Siri itself is a solution in need of a problem, so I don't see how this is really much more so. Plus, I am part of that miniscule minority, so I may find myself actually using Siri if this comes to pass.
    edited November 2018 elijahg
  • Reply 7 of 42
    ajmasajmas Posts: 557member
    lkrupp said:
    This looks like a solution in need of a problem to me. I would think the number of iOS devices, especially iPhones, NOT connected to the Internet is a minuscule minority indeed. My iOS devices are connected 24/7 via WiFi or LTE.
    You sound like someone who has never left their city. The truth is this is useful in a large number of situations, where a signal is not available, including: being out in the middle of nowhere, travelling out of service provider zone, such as another country, and not having a reasonable data plan. 

    When travelling out of country I am often having to hunt WiFi, simply because my providers international roaming options are so expensive. This is where offline services are handy. 
    edited November 2018 space2001elijahgaaronsullivanwatto_cobrajony0
  • Reply 8 of 42
    In other news... water is wet.

    It’s very much a given that Apple’s been researching and experimenting with this for quite some time. If nothing else because it’s cheaper to do as much processing as possible client-side.
    elijahgwatto_cobrajony0
  • Reply 9 of 42
    mike1mike1 Posts: 1,899member
    lkrupp said:
    This looks like a solution in need of a problem to me. I would think the number of iOS devices, especially iPhones, NOT connected to the Internet is a minuscule minority indeed. My iOS devices are connected 24/7 via WiFi or LTE.
    As an example, my connection goes from great to nil during my commute. The fact that I cannot use Siri to play a song that's on my phone seems crazy. Should also be able to dictate a text that gets sent as soon as there is a connection. Or how about let me set a reminder. That stays on the phone. Wish I lived in a world where I always had a fast internet connection. Simply doesn't exist for me.
    elijahgFatmanwatto_cobrajony0
  • Reply 10 of 42

    I'm very happy to hear this...  I drive rurally to/from work and often have points of weak data, although now far less often than when Siri was first released.  For ages I've wanted Siri to at least default to the original Voice Control and try the request again if a network connection wasn't available, as now she just hangs and eventually gives up.

    In fact in the early days I turned it off in the car in favour of Voice Control so I could do simple tasks consistently like voice dialling while I drove (long before Hey Siri, though).  It wasn't until 3G/4G coverage improved in my area that I went Siri full time, but I still have trouble at a few points on my route even today.  It's pretty frustrating when iPhone X’s Siri can't complete a basic command that my 3GS could back in 2009.  This is a welcome change!

    elijahgbaederboywatto_cobrajony0
  • Reply 11 of 42
    this would be awesome combined with giving siri functionality to control certain aspects of the user interface etc. and it would be a great differanciator towards googles assistant. I hate siri being useless when I am in bed, flight mode activated and just want to tell it to start playing, turn on/off lights, etc. only to be presented with "siri not available"
    elijahgwatto_cobra
  • Reply 12 of 42
    hodar said:
    Try asking Siri a very basic question, like "How old am I", or "When will I be 59 1/2 years old".  Siri will suggest websites; which is asinine.
    Hmmm, when I ask my age, or my wife’s age, Siri gives me the correct response. When I ask, “When will I be 59 1/2 years old?” I get the non-response response of “I’m not sure when, I might have to get back to you about that.”

    So, it appears that Siri is better than you imply. But those are questions I, personally, have no need to ask. The things I use Siri for every day are several and work great. That said, I would still appreciate an off-line mode. 

    StrangeDaysaaronsullivancharlesgreswatto_cobra
  • Reply 13 of 42
    hodar said:
    So, if you think Siri is stupid now, just wait.

    Try asking Siri a very basic question, like "How old am I", or "When will I be 59 1/2 years old".  Siri will suggest websites; which is asinine.  Siri knows my birthday, anniversary, the birthday of my kids, friends and neighbors (as long as I have the information in my contacts); but is unable to use that date, and a calendar to answer very basic questions.  Voice recognition on Siri has improved - but Siri's ability to do anything useful with that information is almost 5 years behind both Google, and Amazon - which is pathetic - since Apple practically invented a useful Virtual Assistant.

    Now, between Cortana, Echo and Google Echo - Siri is pretty much a dead last competitor; when it should be the BEST - by far.
    If you don't know when will you be 59 1/2 then Siri is of no use to you. I don't want Apple scientists waste their time with programming such stupid questions into Siri. Carry an Echo in your pocket.
    edited November 2018 elijahgwatto_cobra
  • Reply 14 of 42
    lkrupp said:
    This looks like a solution in need of a problem to me. I would think the number of iOS devices, especially iPhones, NOT connected to the Internet is a minuscule minority indeed. My iOS devices are connected 24/7 via WiFi or LTE.
    As usual you’re in the “I’ve never experienced that problem, so it’s not a problem” brigade. I get it often, and I’m surprised Apple’s not fixed this long ago. It’s incredibly annoying when using Siri on my watch with no idea of the signal level for me to speak, and wait 15 seconds staring at “just a moment...” only for it to say it’s not connected to the internet. It may be the case that I’ve got 4G but it’s too busy to service my request in a reasonable time. 

    The vast majority of tasks I (attempt to) ask Siri to do don’t need the internet - partly because answers requiring research and thus the internet are usually wrong. My requests usually involve adding calendar appointments, reminders or unit conversions. 
    watto_cobraBigDann
  • Reply 15 of 42
    Didn’t the original Siri due offline processing before Apple purchases the company? Glad to see they are breinging back some of the original functionality
    watto_cobra
  • Reply 16 of 42
    hodar said:
    So, if you think Siri is stupid now, just wait.

    Try asking Siri a very basic question, like "How old am I", or "When will I be 59 1/2 years old".  Siri will suggest websites; which is asinine.  Siri knows my birthday, anniversary, the birthday of my kids, friends and neighbors (as long as I have the information in my contacts); but is unable to use that date, and a calendar to answer very basic questions.  Voice recognition on Siri has improved - but Siri's ability to do anything useful with that information is almost 5 years behind both Google, and Amazon - which is pathetic - since Apple practically invented a useful Virtual Assistant.

    Now, between Cortana, Echo and Google Echo - Siri is pretty much a dead last competitor; when it should be the BEST - by far.
    Exactly. What you say should be obvious to anyone. And of course critics are missing the point. It's not about the couple examples you mentioned, its the implication of those examples, extrapolated to so many other scenarios, showing that siri is incredibly stupid, and doesn't use basic information or logic that should be easily accessible to it. A similar example that I remember from my own experience (there are so many that I don't remember as well), was siri giving me directions to an address that is hours away based on one misrepresented character in dictation, even when the correct address is in my gd iphone contacts.


  • Reply 17 of 42
    "Apple wants Siri to become more useful to users.."

    Can we just stop right there? How many little-used features need to be added before we get the fucking basic, rudimentary functionality and context awareness that was promised at siri's launch 6 years ago???
  • Reply 18 of 42
    hodar said:
    So, if you think Siri is stupid now, just wait.

    Try asking Siri a very basic question, like "How old am I", or "When will I be 59 1/2 years old".  Siri will suggest websites; which is asinine.  Siri knows my birthday, anniversary, the birthday of my kids, friends and neighbors (as long as I have the information in my contacts); but is unable to use that date, and a calendar to answer very basic questions.  Voice recognition on Siri has improved - but Siri's ability to do anything useful with that information is almost 5 years behind both Google, and Amazon - which is pathetic - since Apple practically invented a useful Virtual Assistant.

    Now, between Cortana, Echo and Google Echo - Siri is pretty much a dead last competitor; when it should be the BEST - by far.
    Bogus. Siri tells me how old I am just fine. So did you just make that up?

    I do not believe Google and Amazon’s assistants are five years more advanced, either. Articles I’ve read peg them each at different but comparable levels of suckitutde. 


    edited November 2018 charlesgreswatto_cobra
  • Reply 19 of 42
    hodar said:
    So, if you think Siri is stupid now, just wait.

    Try asking Siri a very basic question, like "How old am I", or "When will I be 59 1/2 years old".  Siri will suggest websites; which is asinine.  Siri knows my birthday, anniversary, the birthday of my kids, friends and neighbors (as long as I have the information in my contacts); but is unable to use that date, and a calendar to answer very basic questions.  Voice recognition on Siri has improved - but Siri's ability to do anything useful with that information is almost 5 years behind both Google, and Amazon - which is pathetic - since Apple practically invented a useful Virtual Assistant.

    Now, between Cortana, Echo and Google Echo - Siri is pretty much a dead last competitor; when it should be the BEST - by far.
    Exactly. What you say should be obvious to anyone. And of course critics are missing the point. It's not about the couple examples you mentioned, its the implication of those examples, extrapolated to so many other scenarios, showing that siri is incredibly stupid, and doesn't use basic information or logic that should be easily accessible to it. A similar example that I remember from my own experience (there are so many that I don't remember as well), was siri giving me directions to an address that is hours away based on one misrepresented character in dictation, even when the correct address is in my gd iphone contacts.


    Wow, you are so serious about that "machine intelligence" thing !.. Sorry bro, machines will never "know" anything by themselves. What you call "intelligent assistant" is just a vocal layer in the user interface, nothing more. Pop-culture beliefs and urban legends cannot guide technology, math and engineering can...
    StrangeDayswatto_cobra
  • Reply 20 of 42
    "Apple wants Siri to become more useful to users.."

    Can we just stop right there? How many little-used features need to be added before we get the fucking basic, rudimentary functionality and context awareness that was promised at siri's launch 6 years ago???
    Apple has promised nothing. They have enough human intelligence to not behave with bold claims and manners in such a controversial subject as AI. Yet they provided a phone that recognizes your face. Just consider how many years had to pass for that and imagine how many years have to pass to get a phone that recognizes not just your face but also your habits.
    edited November 2018 watto_cobra
Sign In or Register to comment.