Siri improved small business name recognition by using local language models
Improvements to the way Siri recognizes names of small businesses and local points of interest are due to the use of language models designed for specific locations, Apple's Machine Learning Journal reveals, helping the virtual assistant understand local names for nearby places.

Virtual assistants like Siri are easily capable of understanding the name of prominent businesses and chains, such as supermarkets and restaurant franchises, writes the Siri Speech Recognition Team, with queries concerning lesser-known or regional businesses tending to provide less accurate results. In automatic speech recognition systems (ASR), the team notes this is a "known performance bottleneck" for accuracy, with those further along the long tail of a frequency distribution less likely to be correctly identified.
Apple attempted to improve this for Siri by taking into account the user's location in queries. There would also be two different types of model being used, with a general language model (LM) working alongside a geolocation-based language model (Geo-LM), with the latter becoming more useful if the user is in its coverage area.
ASR systems typically comprise of two components, consisting of an acoustic model that analyzes the properties of speech alongside the language model that analyzes the word usage. Apple noted that the system didn't adequately represent words and names for local points of interest and how they were pronounced, with the more obscure names and combinations also appearing at a very low frequency in the LM training data.
The low frequency means that, in a general LM, the local business name is less likely to be picked up compared to another location, word, or phrase.

In Apple's solution, it defined a number of geographic regions covering most of the United States, producing a Geo-LM for each area. These local versions are used depending on the user's location, though if the user is outside all defined regions or Location Services are disabled, the general LM is used instead.
There are 169 Geo-LM areas for the U.S, based on combined statistical areas defined by the U.S. Census Bureau, covering approximately 80 percent of the population. Each area consists of "adjacent metropolitan areas that are economically and socially linked," measured by commuting patterns.
In Apple's testing, there was no real change in accuracy for general queries, but there was a relative error reduction of 18.7 percent for point of interest-based searches between general LM and Geo-LM usage. In point of interest tests in eight U.S. metropolitan regions, the relative error reduction between general LM and Geo-LM increased, with the localized version performing better by between 41.9 percent and 48.4 percent.
Apple suggests that, because of the limited impact on system speed, the regional coverage of Geo-LM still has room for improvement, but a general language model will be here to stay. "It is essential to continue providing a global Geo-LM in addition to regional LMs," writes Apple, "so that ASR can handle long-distance queries and cases with users located outside supported regions."
International expansion of the program could also occur to languages other than U.S. English, with Apple noting "The method and system proposed here are language independent."
Apple still has some way to go to catch up to Google's level of accuracy for virtual assistants. A July group test revealed Siri has improved its accuracy considerably over the last year to 78.5 percent, as well as increasing its comprehension of queries to close to 100 percent, but under the same test, the Google Assistant achieved an accuracy of 85.5 percent.

Virtual assistants like Siri are easily capable of understanding the name of prominent businesses and chains, such as supermarkets and restaurant franchises, writes the Siri Speech Recognition Team, with queries concerning lesser-known or regional businesses tending to provide less accurate results. In automatic speech recognition systems (ASR), the team notes this is a "known performance bottleneck" for accuracy, with those further along the long tail of a frequency distribution less likely to be correctly identified.
Apple attempted to improve this for Siri by taking into account the user's location in queries. There would also be two different types of model being used, with a general language model (LM) working alongside a geolocation-based language model (Geo-LM), with the latter becoming more useful if the user is in its coverage area.
ASR systems typically comprise of two components, consisting of an acoustic model that analyzes the properties of speech alongside the language model that analyzes the word usage. Apple noted that the system didn't adequately represent words and names for local points of interest and how they were pronounced, with the more obscure names and combinations also appearing at a very low frequency in the LM training data.
The low frequency means that, in a general LM, the local business name is less likely to be picked up compared to another location, word, or phrase.

In Apple's solution, it defined a number of geographic regions covering most of the United States, producing a Geo-LM for each area. These local versions are used depending on the user's location, though if the user is outside all defined regions or Location Services are disabled, the general LM is used instead.
There are 169 Geo-LM areas for the U.S, based on combined statistical areas defined by the U.S. Census Bureau, covering approximately 80 percent of the population. Each area consists of "adjacent metropolitan areas that are economically and socially linked," measured by commuting patterns.
In Apple's testing, there was no real change in accuracy for general queries, but there was a relative error reduction of 18.7 percent for point of interest-based searches between general LM and Geo-LM usage. In point of interest tests in eight U.S. metropolitan regions, the relative error reduction between general LM and Geo-LM increased, with the localized version performing better by between 41.9 percent and 48.4 percent.
Apple suggests that, because of the limited impact on system speed, the regional coverage of Geo-LM still has room for improvement, but a general language model will be here to stay. "It is essential to continue providing a global Geo-LM in addition to regional LMs," writes Apple, "so that ASR can handle long-distance queries and cases with users located outside supported regions."
International expansion of the program could also occur to languages other than U.S. English, with Apple noting "The method and system proposed here are language independent."
Apple still has some way to go to catch up to Google's level of accuracy for virtual assistants. A July group test revealed Siri has improved its accuracy considerably over the last year to 78.5 percent, as well as increasing its comprehension of queries to close to 100 percent, but under the same test, the Google Assistant achieved an accuracy of 85.5 percent.
Comments
about Siri in general, there’s a reason why Siri lags. There is a difference in the way Apple, Google, Amazon and Microsoft are doing this. The others are using the technique called Deep Learning for this. Apple is using a technique, the name of which just popped away from me, that works very differently.
deep learning, which I think is misnamed, uses an AI technology that requires that vast amounts of information are force fed to it. The AI then figures out links, both mechanically and logically related, and can do some minor reasoning. The AI itself doesn’t have much depth to it. It also requires that the work be done on remote server farms, the way Siri worked in the beginning. Deep learning is being questioned as a l9ng term technology, with experts stating that it won’t be able to advance much further. An article recently stated that the end was pretty close. Google is attempting to modify the was it works for them.
apple isn’t working that way. Much of what Siri does now is done one the device itself. This is why Apple has been working on the on-chip AI so hard. Apple’s method doesn’t rely on massive amounts of data, particularly personal data. The AI is actually deeper than than that in deep eating. But, it’s a much harder technology to master. Because of that, Siri lags. But as we can see, Siri is improving faster. In another two or three years, it should begin to regain the lead. It’s been said that Apple’s ongoing persistence with privacy is holding Siri back, and it’s true. But Apple’s work is far less dependent on personal data, and if they can pull this off, then they’ll have a big advantage.
Clearly there’s still work to be done.
Still, that’s moderately cumbersome to say and it seems like an easy thing for the Siri team to overcome. Alternate spellings are fairly common in business names and I certainly don’t want to have to be that explicit if I’m asking for directions or a phone number or whatever.
The unfortunate thing is the owner was always complaining to me that Siri couldn’t provide the phone number to her shop, but if she asked google it did. She wasn’t asking Google for “K dash 9”, but it still managed to figure out what she needed. She actually switched to an Android phone because of that.
at any rate, I think it’s going to take another five years before things settle down.
The truth is that products only get improved when they are hindering sales. SIRI and Maps are not hindering sales.
Could SIRI be better? Of course, but so could Google's 85% accurate voice assistant. The delta between 78.5% accuracy and 85% accuracy is 6.5 errors out of 100 attempts. This is not a deal breaker when you consider the alternative: buying an Android smartphone with Google Voice vs buying an iPhone with SIRI.
Will SIRI and Maps get better? Undoubtedly, but improvements from 78.5% to 85% are hardly noticeable in the real world, and even less so as accuracy goes above 85%.
i notice that I didn’t catch the typo “deep eating”. Well, I guess it’s right, because it does eat a lot of data.
You don’t agree that Apple has to improve this for the almost 65% of their customers who aren’t in the USA? So it’s ok if we get much better service from Apple’s software than anyone else? That’s a very atrange bit of thinking.
usually I don’t say that Apple must, or should do anything. I just say that I think they should do it, which is different. But for this, yes, they need to do it if they’re going to bring the percentage of users up, which we know they want to do.
its not a solution that I have to present here. I don’t even know why you would say that. All that needs to be done is to bring the software and service up to our level. Apple knows how to do this. It’s not a solution they need. They just have to hire the people needed. When they began to improve maps, they said they hired thousands of people to do it. But most of their efforts center here. They’ve stated that they want to have the best mapping service of anyone. They’ve said the same thing about Siri. Now, they just have to do it. No new solutions required other than effort and some of that cash pile. The difference between 85% and 78.5% is noticeable. Going from 78.5 to 85 is 8% and if you get 8 more wrong answers out of a hundred, you will notice it.
how does Apple manage this? People don’t stay in one place.