Investigation into Siri reveals how the iPhone 4S service talks to Apple
A mobile developer investigating Apple's Siri voice assistant has dissected the service's protocol to develop tools for playing with the service outside of the iPhone 4S.
Testing by Applidium notes that iPhone 4S uses standard HTTPS network requests to communicate with Apple's servers, but sends data using an "ACE" command rather than regular web GET requests.
Each Siri request an iPhone 4S makes also involves a unique host identifier that appears to be based on the hardware UUID, preventing unauthorized devices from sending requests to Apple's servers.
Applidium reports some success in copying an iPhone 4S host identifier into requests sent from other devices, including a test Mac environment. By examining how iPhone 4S packages speech recognition requests, the developer was able to send a similarly packaged request and obtain a correct response.
The testing proves that Siri sends raw audio captures of the user's voice, compressed with the Speex audio codec optimized for VoIP. Previously, it had been speculated that iPhone 4S was performing preprocessing of the audio and sending only the results to Apple's servers.
While Siri may perform other preprocessing tasks that use the additional horsepower of the iPhone 4S, Applidium's discovery indicates that any iPhone should be able to support at least Siri's basic voice recognition features, although Apple has indicated that it has no plans to release such capabilities for earlier iOS 5 models, including iPhone 4 and iPhone 3GS.
Both models can support third party speech-to-text services, but Apple offers no way in iOS to integrate such third party services into any app system wide, meaning that users have to dictate into one app and then copy and paste the results elsewhere. Both Android and Windows Phone 7 offer system wide, integrated voice recognition features.
Siri, however, goes far beyond simple voice recognition. Rather than just converting audio to text, Siri evaluates the meaning of requests and maintains a understanding of the user's relationships with specific contacts and a contextual session of the location and other details of a request.
So far, Applidium's investigation has revealed that Siri packages requests in compressed property lists, but further exploration of the protocol is hampered by a number of issues, including the complexity of requests, the fact that they are tied to a hardware key, and that they are subject to change.
Apple could at any time stop responding to a given hardware key were it to suspect that it was being used to exploit its servers; additionally, because the Siri service is proprietary to Apple, the company can change how it transmits data rather quickly by simply sending out a client update.
Applidium says "anyone could now write an Android app that uses the real Siri! Or use Siri on an iPad!" However, in order to access Siri at all, a user would have to sniff out the unique user key of an actual iPhone 4S, and then reuse that key until it expired or was blocked by Apple.
Apple has been perfecting its Siri service as a "beta" feature exclusive to the iPhone 4S, but has experience some downtime in ramping up services to accommodate the demands of millions of users who have rushed to buy the latest iPhone model. The new service also makes use of hardware unique to the iPhone 4S.
It is expected that Siri will eventually find its way to new models of the iPod touch, iPad and perhaps even Macs, with some speculating that it could eventually server as a living room interface employed by Apple TV, doing away with the need for a button or touch-based remote control for navigating television programing.
Testing by Applidium notes that iPhone 4S uses standard HTTPS network requests to communicate with Apple's servers, but sends data using an "ACE" command rather than regular web GET requests.
Each Siri request an iPhone 4S makes also involves a unique host identifier that appears to be based on the hardware UUID, preventing unauthorized devices from sending requests to Apple's servers.
Applidium reports some success in copying an iPhone 4S host identifier into requests sent from other devices, including a test Mac environment. By examining how iPhone 4S packages speech recognition requests, the developer was able to send a similarly packaged request and obtain a correct response.
The testing proves that Siri sends raw audio captures of the user's voice, compressed with the Speex audio codec optimized for VoIP. Previously, it had been speculated that iPhone 4S was performing preprocessing of the audio and sending only the results to Apple's servers.
While Siri may perform other preprocessing tasks that use the additional horsepower of the iPhone 4S, Applidium's discovery indicates that any iPhone should be able to support at least Siri's basic voice recognition features, although Apple has indicated that it has no plans to release such capabilities for earlier iOS 5 models, including iPhone 4 and iPhone 3GS.
Both models can support third party speech-to-text services, but Apple offers no way in iOS to integrate such third party services into any app system wide, meaning that users have to dictate into one app and then copy and paste the results elsewhere. Both Android and Windows Phone 7 offer system wide, integrated voice recognition features.
Siri, however, goes far beyond simple voice recognition. Rather than just converting audio to text, Siri evaluates the meaning of requests and maintains a understanding of the user's relationships with specific contacts and a contextual session of the location and other details of a request.
So far, Applidium's investigation has revealed that Siri packages requests in compressed property lists, but further exploration of the protocol is hampered by a number of issues, including the complexity of requests, the fact that they are tied to a hardware key, and that they are subject to change.
Apple could at any time stop responding to a given hardware key were it to suspect that it was being used to exploit its servers; additionally, because the Siri service is proprietary to Apple, the company can change how it transmits data rather quickly by simply sending out a client update.
Applidium says "anyone could now write an Android app that uses the real Siri! Or use Siri on an iPad!" However, in order to access Siri at all, a user would have to sniff out the unique user key of an actual iPhone 4S, and then reuse that key until it expired or was blocked by Apple.
Apple has been perfecting its Siri service as a "beta" feature exclusive to the iPhone 4S, but has experience some downtime in ramping up services to accommodate the demands of millions of users who have rushed to buy the latest iPhone model. The new service also makes use of hardware unique to the iPhone 4S.
It is expected that Siri will eventually find its way to new models of the iPod touch, iPad and perhaps even Macs, with some speculating that it could eventually server as a living room interface employed by Apple TV, doing away with the need for a button or touch-based remote control for navigating television programing.
Comments
Siri can work with every other network-based device in the world yet Apple only allows the iPhone 4S to use it¡ OMG¡ OMG¡
Watch Apple have to shut the entire service down from the load this reverse engineering could potentially cause, as the servers were designed to handle the amount of 4S' sold.
Watch Apple have to shut the entire service down from the load this reverse engineering could potentially cause, as the servers were designed to handle the amount of 4S' sold.
Yes, what a great public service this nerd is providing. Why can't people just let it be?
Watch Apple have to shut the entire service down from the load this reverse engineering could potentially cause, as the servers were designed to handle the amount of 4S' sold.
Why? Apple allready has a filter for valid devices. If they believe a device is invalid they can either blacklist it or do some other identification. My guess is that they will blacklist and use legal tactics to make sure this doesn't last.
Yes, what a great public service this nerd is providing. Why can't people just let it be?
Why can't I be curious on how Siri does work? As an avid programmer, I love to just expand my knowledge of technology and I especially love to see how Apple's technology for iOS works, including Siri; there's no harm in knowing.
Besides, this information is pretty basic, just because it was explained how it works doesn't mean that there are a plethora of people at the gates with this knowledge now able to mimic Siri requests. Nothing will really change.
Watch Apple have to shut the entire service down from the load this reverse engineering could potentially cause, as the servers were designed to handle the amount of 4S' sold.
Nah... Apple will just send out a patch that lets Siri know whether to answer an iPhone 4s device in the normal Siri voice or if it is not an Apple device, code Siri to respond in that generic male computer voice "Does not compute! Does not compute!"
Oh, and anyone that tinkers with their none Apple iPhone 4s by fooling Siri into thinking it's an iPhone 4s, well I heard Apple legal just got a box of 1,000,000,000 business cards...
Hello, this is Apple legal calling!
/
/
/
Siri should be on iPod nano.
This can't happen until storage is large enough cheaply enough to house all of the server-side software on every device.
Imagine being able to place and receive calls on a watch-sized nano.
Oh, so not an iPod at all, an iPhone with a 1" screen. Somehow I don't think that'll work out.
And sell it for $299.
Battery life: 30 minutes!
Battery life: 30 minutes!
It can go two ways. You can have these newfangled new concepts fairly soon with no realistic battery life, or have them with realistic battery life in some future that makes the whole idea too distant to be worth conceiving. It's like me saying I want a tablet that is thinner and light with 10x the battery life. I'm sure that will happen, but not for a very, very long time.
http://www.youtube.com/watch?v=C_r-30oLdH0
Interesting.
"The power of this technology is NOT in voice recognition - it is being able to UNDERSTAND complex statements and deduce MEANING."
This can't happen until storage is large enough cheaply enough to house all of the server-side software on every device.
Not to mention that the Nano has no network connectivity.
Yes, what a great public service this nerd is providing. Why can't people just let it be?
Here?s to the crazy ones. The misfits. The rebels. The troublemakers. The round pegs in the square holes.
Here's a video explanation of Siri I found on YouTube.
http://www.youtube.com/watch?v=C_r-30oLdH0
Interesting.
"The power of this technology is NOT in voice recognition - it is being able to UNDERSTAND complex statements and deduce MEANING."
Exactly!
The speech recognition is provided by Nuance and although complex and challenging, speech recognition is fairly well understood. Siri provides natural language processing with contextual, semantic and syntactical awareness. Siri is capable of far more than the functionality currently available on iPhone 4S. Essentially, Siri could integrate with any number and variety of products and services to extend the capabilities to order movie tickets, restaurant reservations or requesting taxis. In fact, Siri was previously capable of doing all those activities.
Not to mention that the Nano has no network connectivity.
That's EXACTLY what the other poster WAS mentioning.
No network connectivity => all processing happens locally => local storage and processing power must increase dramatically.
Thompson
Nowadays, these forums are full of people who, whenever someone?whether a developer like this, or another company, or a security researcher?finds something interesting, something novel, something reverse-engineered, something flawed?are quick to shut it down, declare it out of bounds, say it's useless and call for Apple to block it.
People, this is technology! It's ones and zeroes. Examine them, decode them, rebuild them, restructure them, bask in their ingenuity! A savvy programmer tinkers with Siri and gets it working on another device, figures out how requests are sent, decodes the authentication mechanism and finds a workaround. This is brilliant, and it illuminates the way the technology we love works.
Maybe it's just your desire to preserve the aura of magic that Jobs so cunningly cultivated, but when you call for Apple Legal to break out the blocklists (as if a tiny spike in traffic on one iPhone 4S UUID would even attract their notice), you're not helping Apple, you're hurting the kind of people who get excited about its technologies and want to develop them further, even maybe if they have different ideas about how they are to be used than those in Apple's engineering department.
Apple (and mainly Jobs) created this mindset where things are locked down, where everything is packaged so neatly and cleanly and does what it's supposed to do well that many of the Mac faithful have been sucked into believing that Apple's way is the only way. This used to be the platform that called for "thinking differently" but in some Apple fans' calls to follow Apple in every situation, I think we run the risk of becoming merely sheep. Now that Tim Cook is in charge, hopefully an environment more open to free-thinking can be cultivated.
I don't mean any personal offense with this post, and I'm not calling out anyone in particular, but sometimes the lack of imagination and the rush to defense makes me sad. This coming from someone who's been using Mac since elementary school (1994).
Siri should be on iPod nano. That would totally create a new market segment which Apple's competitors cannot even match. Imagine being able to place and receive calls on a watch-sized nano. Make appointments. Have it read your email etc. Track your fitness like that Jawbone UP thing. And sell it for $299.
And it could be clipped on as a watch. It will of course take another round of miniaturizing to fit in bluetooth, WiFi, GSM and GPS, but it will be the ultimate communication device.
J.
That's EXACTLY what the other poster WAS mentioning.
No network connectivity => all processing happens locally => local storage and processing power must increase dramatically.
Thompson
Once upon a time, the iPod was the size of a deck of cards, held 5GB, had a click wheel, no video etc.
In 1997, a roomful of 10,000 Pentium Pros drew 500 KW of power and required another 500 KW just for cooling. Ten years later, Intel had got that down to the size of a fingernail drawing just 62W.
I don't think it's that far-fetched. All the nano needs is network connectivity. Without a screen, battery demands are far less.