iOS 10 cracks open Apple's Siri voice assistance to third party apps
Two years ago, Apple introduced App Extensions to enable third party developers to add system-wide features using a secure new architecture. This year, iOS 10 brings App Extensions to Siri, enabling developers to teach Siri how to expand its capabilities.
Since its release back in 2011 on iPhone 4s under iOS 5, Siri's core abilities to perform system tasks--such as setting alarms, finding contacts, checking the weather and getting directions via Maps (and integrating with Apple's partners including Yelp and Wolfram Alpha)--have incrementally expanded to add support for a variety of new functions--sport scores, restaurant reservations and movie showtimes in iOS 6; searching Wikipedia, Bing or Twitter in iOS 7, integrating with CarPlay and HomeKit devices in iOS 8 and navigating Apple Music and Apple Watch last year.
Now, however, Apple is using its App Extension mechanism to allow third party developers to build new support for Siri into their apps by registering "domains" and "intents" they can handle.
Like other Extension Points, Apple has created a limited set of the most common and useful third party "domains" that make sense to invoke by voice, creating a well known foundation it can build upon going forward.
Apps can now register within one of Siri's domains of functionality: audio or voice calling from third party VoIP apps; text messaging; payments; searching Photos; managing workouts in third party apps; booking rides in car sharing services; and new support for automotive control features in vehicles that use CarPlay.
It's noteworthy that Apple has focused Siri's new extensible capabilities on the most used and most attractive features in current voice services. Two studies conducted by Ben Bajarin, a principle analyst at Creative Strategies, showed that the top use cases for voice assistants include calling or texting someone.
Prior to WWDC, rumors suggested that Apple would be adding direct Apple Pay support to either Messages or perhaps Siri. Instead, the company demonstrated a partnership with Square for Siri-invoked payments. Incidentally, that's the same person-to-person payment service Snapchat teamed up with to create Snapcash, a deal that was hailed as being innovative.
It's possible that Apple will add its own Apple Pay support to Siri in the future, but it may also be the case that there's currently no real money in sending money for free; Apple may be content to allow its third party app partners to provide such payment services. By opening up the mechanism for Siri payments, virtually any app can register the ability to send or receive money.
That's also essentiality the case with ride sharing services like Uber and Lyft, which currently subsidize riders with venture capital in the hope of building a business that will someday be profitable (perhaps as soon as drivers--the most expensive element--can be replaced with self driving car robots).
Three remaining Siri domains relate to strategic expansions of Apple's ecosystem: apps that tap into Photos; Workout apps that expand upon the core motion features of iPhones and Apple Watch to track workouts; and CarPlay integration.
In each of these cases, Siri continues to manage all of the user interface and voice recognition tasks. When it hears a request for a domain of functionality involving an installed app supplying a Siri App Extension, it can pass the request to that app even if the app isn't currently running.
Within each domain, Siri interprets user "intents"--what it understands the user wants to accomplish--and passes this to the app to handle. For example, a user may ask to initiate a new audio or video call to a given contact using a specific app such as WhatsApp or Skype (as opposed to Apple's own default FaceTime or Phone services).
A user might ask to send or request money using a payment service like Square, or ask a workout app to create a new workout or pause, resume, end or cancel the current workout.
In a supported CarPlay vehicle, Siri can pass commands to the car to change the audio playback source, change the radio station, change climate control settings, set the seat temperature or control the defroster.
In some cases, an App Extension handler needs to provide a response to an intent passed from Siri in order to confirm the task with the user before it begins, particularly when a financial transaction or some other irreversible action is involved (below).
Intents are outlined by Apple as supported, extendable actions within a given domain. This architecture enables Apple to build out increasingly sophisticated realms of new domains populated with rich arrays of new intents going forward.
Developers can also register vocabulary for Siri to understand, such as brand names or features. A ride sharing app like Uber might register terms such as a requested vehicle type ("UberX") so users can request that type of service from Siri by name.
Bajarin's previously mentioned study noted that "more consumers across the spectrum of operating systems (iOS, Android and Windows) have used Siri than any other voice UI." He credited iPad with assisting here, because many Android, Windows PC and feature phone users have iPads, even if they don't have an iPhone.
Not only is Siri used more often that any other voice assistant, but it's also supported across far more languages and regions--currently more than 30. That means developers will have to account for requests made in other languages. However, Apple also makes this easier to accomplish by handling much of the language-specific work on its own, allowing developers to focus on functionality in their apps rather than trying to become voice services experts in every major language where their app may be used.
At WWDC, Apple noted that Siri now logs two billion requests per week. That gives the company pretty clear insight into how users expect to use Siri. As it works to perfect initial support for App Extensions across the first seven primary Siri domains, it will also get new feedback from developers working on apps or new projects that would also like to tap into Siri.
SiriKit isn't the only framework to make use of new App Extensions, or even the Intents architecture Siri uses to pass user requests to third party apps. The following segment will look at another Extension Point Apple is adding in iOS 10 related to Maps.
Siri follows a familiar route to third party extensions
Since its release back in 2011 on iPhone 4s under iOS 5, Siri's core abilities to perform system tasks--such as setting alarms, finding contacts, checking the weather and getting directions via Maps (and integrating with Apple's partners including Yelp and Wolfram Alpha)--have incrementally expanded to add support for a variety of new functions--sport scores, restaurant reservations and movie showtimes in iOS 6; searching Wikipedia, Bing or Twitter in iOS 7, integrating with CarPlay and HomeKit devices in iOS 8 and navigating Apple Music and Apple Watch last year.
Now, however, Apple is using its App Extension mechanism to allow third party developers to build new support for Siri into their apps by registering "domains" and "intents" they can handle.
Like other Extension Points, Apple has created a limited set of the most common and useful third party "domains" that make sense to invoke by voice, creating a well known foundation it can build upon going forward.
Siri Domains
Apps can now register within one of Siri's domains of functionality: audio or voice calling from third party VoIP apps; text messaging; payments; searching Photos; managing workouts in third party apps; booking rides in car sharing services; and new support for automotive control features in vehicles that use CarPlay.
It's noteworthy that Apple has focused Siri's new extensible capabilities on the most used and most attractive features in current voice services. Two studies conducted by Ben Bajarin, a principle analyst at Creative Strategies, showed that the top use cases for voice assistants include calling or texting someone.
Prior to WWDC, rumors suggested that Apple would be adding direct Apple Pay support to either Messages or perhaps Siri. Instead, the company demonstrated a partnership with Square for Siri-invoked payments. Incidentally, that's the same person-to-person payment service Snapchat teamed up with to create Snapcash, a deal that was hailed as being innovative.
It's possible that Apple will add its own Apple Pay support to Siri in the future, but it may also be the case that there's currently no real money in sending money for free; Apple may be content to allow its third party app partners to provide such payment services. By opening up the mechanism for Siri payments, virtually any app can register the ability to send or receive money.
That's also essentiality the case with ride sharing services like Uber and Lyft, which currently subsidize riders with venture capital in the hope of building a business that will someday be profitable (perhaps as soon as drivers--the most expensive element--can be replaced with self driving car robots).
Three remaining Siri domains relate to strategic expansions of Apple's ecosystem: apps that tap into Photos; Workout apps that expand upon the core motion features of iPhones and Apple Watch to track workouts; and CarPlay integration.
In each of these cases, Siri continues to manage all of the user interface and voice recognition tasks. When it hears a request for a domain of functionality involving an installed app supplying a Siri App Extension, it can pass the request to that app even if the app isn't currently running.
Siri Intents
Within each domain, Siri interprets user "intents"--what it understands the user wants to accomplish--and passes this to the app to handle. For example, a user may ask to initiate a new audio or video call to a given contact using a specific app such as WhatsApp or Skype (as opposed to Apple's own default FaceTime or Phone services).
A user might ask to send or request money using a payment service like Square, or ask a workout app to create a new workout or pause, resume, end or cancel the current workout.
In a supported CarPlay vehicle, Siri can pass commands to the car to change the audio playback source, change the radio station, change climate control settings, set the seat temperature or control the defroster.
In some cases, an App Extension handler needs to provide a response to an intent passed from Siri in order to confirm the task with the user before it begins, particularly when a financial transaction or some other irreversible action is involved (below).
Intents are outlined by Apple as supported, extendable actions within a given domain. This architecture enables Apple to build out increasingly sophisticated realms of new domains populated with rich arrays of new intents going forward.
Developers can also register vocabulary for Siri to understand, such as brand names or features. A ride sharing app like Uber might register terms such as a requested vehicle type ("UberX") so users can request that type of service from Siri by name.
Apple's Siri "the most used voice-based user interface"
Bajarin's previously mentioned study noted that "more consumers across the spectrum of operating systems (iOS, Android and Windows) have used Siri than any other voice UI." He credited iPad with assisting here, because many Android, Windows PC and feature phone users have iPads, even if they don't have an iPhone.
For everyone thinking Apple isn't getting usage data to make Siri smarter, Siri is by far the most used voice assistant than any out there.
-- Ben Bajarin (@BenBajarin)
Not only is Siri used more often that any other voice assistant, but it's also supported across far more languages and regions--currently more than 30. That means developers will have to account for requests made in other languages. However, Apple also makes this easier to accomplish by handling much of the language-specific work on its own, allowing developers to focus on functionality in their apps rather than trying to become voice services experts in every major language where their app may be used.
At WWDC, Apple noted that Siri now logs two billion requests per week. That gives the company pretty clear insight into how users expect to use Siri. As it works to perfect initial support for App Extensions across the first seven primary Siri domains, it will also get new feedback from developers working on apps or new projects that would also like to tap into Siri.
SiriKit isn't the only framework to make use of new App Extensions, or even the Intents architecture Siri uses to pass user requests to third party apps. The following segment will look at another Extension Point Apple is adding in iOS 10 related to Maps.
Comments
Just FYI.
Also can we finally download extensions without having to download an app?
I've been wanting this since iPhone launched. It's stupid to have to download an app just to use its extension. Just have all your extensions accessible in settings and stored in the cloud. (Siri doesn't work without a data connection anyways.) I don't wanna waste space on apps just to ask specific queries.
I would hate to have to download the eBay app just to ask Siri eBay related questions. I wish Apple could just compile all this 3rd party info for us so we can just ask anything we need.
I remember having to download the Swype app and then hiding it just to have the keyboard as an option.
From the description, it doesn't sound like the apps in question will have access to the voice which made the request. Siri will interpret the request and send the intent on to the individual app or extension. The app never gets the actual voiced instructions.