Siri improved small business name recognition by using local language models

Posted:
in iOS
Improvements to the way Siri recognizes names of small businesses and local points of interest are due to the use of language models designed for specific locations, Apple's Machine Learning Journal reveals, helping the virtual assistant understand local names for nearby places.




Virtual assistants like Siri are easily capable of understanding the name of prominent businesses and chains, such as supermarkets and restaurant franchises, writes the Siri Speech Recognition Team, with queries concerning lesser-known or regional businesses tending to provide less accurate results. In automatic speech recognition systems (ASR), the team notes this is a "known performance bottleneck" for accuracy, with those further along the long tail of a frequency distribution less likely to be correctly identified.

Apple attempted to improve this for Siri by taking into account the user's location in queries. There would also be two different types of model being used, with a general language model (LM) working alongside a geolocation-based language model (Geo-LM), with the latter becoming more useful if the user is in its coverage area.

ASR systems typically comprise of two components, consisting of an acoustic model that analyzes the properties of speech alongside the language model that analyzes the word usage. Apple noted that the system didn't adequately represent words and names for local points of interest and how they were pronounced, with the more obscure names and combinations also appearing at a very low frequency in the LM training data.

The low frequency means that, in a general LM, the local business name is less likely to be picked up compared to another location, word, or phrase.




In Apple's solution, it defined a number of geographic regions covering most of the United States, producing a Geo-LM for each area. These local versions are used depending on the user's location, though if the user is outside all defined regions or Location Services are disabled, the general LM is used instead.

There are 169 Geo-LM areas for the U.S, based on combined statistical areas defined by the U.S. Census Bureau, covering approximately 80 percent of the population. Each area consists of "adjacent metropolitan areas that are economically and socially linked," measured by commuting patterns.

In Apple's testing, there was no real change in accuracy for general queries, but there was a relative error reduction of 18.7 percent for point of interest-based searches between general LM and Geo-LM usage. In point of interest tests in eight U.S. metropolitan regions, the relative error reduction between general LM and Geo-LM increased, with the localized version performing better by between 41.9 percent and 48.4 percent.

Apple suggests that, because of the limited impact on system speed, the regional coverage of Geo-LM still has room for improvement, but a general language model will be here to stay. "It is essential to continue providing a global Geo-LM in addition to regional LMs," writes Apple, "so that ASR can handle long-distance queries and cases with users located outside supported regions."

International expansion of the program could also occur to languages other than U.S. English, with Apple noting "The method and system proposed here are language independent."

Apple still has some way to go to catch up to Google's level of accuracy for virtual assistants. A July group test revealed Siri has improved its accuracy considerably over the last year to 78.5 percent, as well as increasing its comprehension of queries to close to 100 percent, but under the same test, the Google Assistant achieved an accuracy of 85.5 percent.

Comments

  • Reply 1 of 19
    melgrossmelgross Posts: 33,510member
    Apple needs to improve the international functions of this and that of Maps. The biggest difference in performance between Google’s software products and Apple’s now are that Google’s are far better covered internationally than Apple’s. I know that international rollouts take some time, but Apple has to do better if we’re going to stop seeing the heavy criticism of Apple over this.

    about Siri in general, there’s a reason why Siri lags. There is a difference in the way Apple, Google, Amazon and Microsoft are doing this. The others are using the technique called Deep Learning for this. Apple is using a technique, the name of which just popped away from me, that works very differently.

    deep learning, which I think is misnamed, uses an AI technology that requires that vast amounts of information are force fed to it. The AI then figures out links, both mechanically and logically related, and can do some minor reasoning. The AI itself doesn’t have much depth to it. It also requires that the work be done on remote server farms, the way Siri worked in the beginning. Deep learning is being questioned as a l9ng term technology, with experts stating that it won’t be able to advance much further. An article recently stated that the end was pretty close. Google is attempting to modify the was it works for them.

    apple isn’t working that way. Much of what Siri does now is done one the device itself. This is why Apple has been working on the on-chip AI so hard. Apple’s method doesn’t rely on massive amounts of data, particularly personal data. The AI is actually deeper than than that in deep eating. But, it’s a much harder technology to master. Because of that, Siri lags. But as we can see, Siri is improving faster. In another two or three years, it should begin to regain the lead. It’s been said that Apple’s ongoing persistence with privacy is holding Siri back, and it’s true. But Apple’s work is far less dependent on personal data, and if they can pull this off, then they’ll have a big advantage.
    edited August 2018 wonkothesanedouglas bailey
  • Reply 2 of 19
    For years Siri has heard me ask for directions to Germaine’s K-9 Kuties and failed, in part, because it heard “Germaine” as “Jermaine”. Now Siri is getting “Germaine” correct but giving me directions to “Canine Cuties”, a business that is in New Jersey and a three hour drive. The business I’m looking for is less than 10 miles from me and when I type it into Maps comes up correctly. 

    Clearly there’s still work to be done. 
    [Deleted User]
  • Reply 3 of 19
    melgrossmelgross Posts: 33,510member
    For years Siri has heard me ask for directions to Germaine’s K-9 Kuties and failed, in part, because it heard “Germaine” as “Jermaine”. Now Siri is getting “Germaine” correct but giving me directions to “Canine Cuties”, a business that is in New Jersey and a three hour drive. The business I’m looking for is less than 10 miles from me and when I type it into Maps comes up correctly. 

    Clearly there’s still work to be done. 
    Just out of curiosity, have you tried being more literal, by saying K-9 as K dash 9?
    watto_cobra
  • Reply 4 of 19
    melgross said:
    For years Siri has heard me ask for directions to Germaine’s K-9 Kuties and failed, in part, because it heard “Germaine” as “Jermaine”. Now Siri is getting “Germaine” correct but giving me directions to “Canine Cuties”, a business that is in New Jersey and a three hour drive. The business I’m looking for is less than 10 miles from me and when I type it into Maps comes up correctly. 

    Clearly there’s still work to be done. 
    Just out of curiosity, have you tried being more literal, by saying K-9 as K dash 9?
    Good catch! I had never tried that but it just worked. I have tried really annunciating the “K” and the “nine” and keeping them separate, but that didn’t help. 

    Still, that’s moderately cumbersome to say and it seems like an easy thing for the Siri team to overcome. Alternate spellings are fairly common in business names and I certainly don’t want to have to be that explicit if I’m asking for directions or a phone number or whatever. 
    edited August 2018
  • Reply 5 of 19
    melgrossmelgross Posts: 33,510member
    melgross said:
    For years Siri has heard me ask for directions to Germaine’s K-9 Kuties and failed, in part, because it heard “Germaine” as “Jermaine”. Now Siri is getting “Germaine” correct but giving me directions to “Canine Cuties”, a business that is in New Jersey and a three hour drive. The business I’m looking for is less than 10 miles from me and when I type it into Maps comes up correctly. 

    Clearly there’s still work to be done. 
    Just out of curiosity, have you tried being more literal, by saying K-9 as K dash 9?
    Good catch! I had never tried that but it just worked. I have tried really annunciating the “K” and the “nine” and keeping them separate, but that didn’t help. 

    Still, that’s moderately cumbersome to say and it seems like an easy thing for the Siri team to overcome. Alternate spellings are fairly common in business names and I certainly don’t want to have to be that explicit if I’m asking for directions or a phone number or whatever. 
    It’s really difficult for people to tell the differences, much less AI. At best, it would find both, and ask you which was correct. I’ve had that happen a number of times. Or you ask for the closest. If you think about it, to an AI, K-9 is exactly that, K dash 9. Sometimes computers force us into being more literal. It can be a good thing.
  • Reply 6 of 19
    melgross said:
    melgross said:
    For years Siri has heard me ask for directions to Germaine’s K-9 Kuties and failed, in part, because it heard “Germaine” as “Jermaine”. Now Siri is getting “Germaine” correct but giving me directions to “Canine Cuties”, a business that is in New Jersey and a three hour drive. The business I’m looking for is less than 10 miles from me and when I type it into Maps comes up correctly. 

    Clearly there’s still work to be done. 
    Just out of curiosity, have you tried being more literal, by saying K-9 as K dash 9?
    Good catch! I had never tried that but it just worked. I have tried really annunciating the “K” and the “nine” and keeping them separate, but that didn’t help. 

    Still, that’s moderately cumbersome to say and it seems like an easy thing for the Siri team to overcome. Alternate spellings are fairly common in business names and I certainly don’t want to have to be that explicit if I’m asking for directions or a phone number or whatever. 
    It’s really difficult for people to tell the differences, much less AI. At best, it would find both, and ask you which was correct. I’ve had that happen a number of times. Or you ask for the closest. If you think about it, to an AI, K-9 is exactly that, K dash 9. Sometimes computers force us into being more literal. It can be a good thing.
    I thought that’s what this article was about, finding the businesses that are close to your location, without you having to specify you want the one 10 minutes away, not the one three hours away. 

    The unfortunate thing is the owner was always complaining to me that Siri couldn’t provide the phone number to her shop, but if she asked google it did. She wasn’t asking Google for “K dash 9”, but it still managed to figure out what she needed. She actually switched to an Android phone because of that. 
    edited August 2018 [Deleted User]
  • Reply 7 of 19
    melgrossmelgross Posts: 33,510member
    melgross said:
    melgross said:
    For years Siri has heard me ask for directions to Germaine’s K-9 Kuties and failed, in part, because it heard “Germaine” as “Jermaine”. Now Siri is getting “Germaine” correct but giving me directions to “Canine Cuties”, a business that is in New Jersey and a three hour drive. The business I’m looking for is less than 10 miles from me and when I type it into Maps comes up correctly. 

    Clearly there’s still work to be done. 
    Just out of curiosity, have you tried being more literal, by saying K-9 as K dash 9?
    Good catch! I had never tried that but it just worked. I have tried really annunciating the “K” and the “nine” and keeping them separate, but that didn’t help. 

    Still, that’s moderately cumbersome to say and it seems like an easy thing for the Siri team to overcome. Alternate spellings are fairly common in business names and I certainly don’t want to have to be that explicit if I’m asking for directions or a phone number or whatever. 
    It’s really difficult for people to tell the differences, much less AI. At best, it would find both, and ask you which was correct. I’ve had that happen a number of times. Or you ask for the closest. If you think about it, to an AI, K-9 is exactly that, K dash 9. Sometimes computers force us into being more literal. It can be a good thing.
    I thought that’s what this article was about, finding the businesses that are close to your location, without you having to specify you want the one 10 minutes away, not the one three hours away. 

    The unfortunate thing is the owner was always complaining to me that Siri couldn’t provide the phone number to her shop, but if she asked google it did. She wasn’t asking Google for “K dash 9”, but it still managed to figure out what she needed. She actually switched to an Android phone because of that. 
    I don’t know, but I’ve tried Google a number of times for things, and it goofed up too. Neither is perfect. It also depends on where. Possiblely she’s asking from a different place than you are. That could account for the difference.

    at any rate, I think it’s going to take another five years before things settle down.
  • Reply 8 of 19
    croprcropr Posts: 1,122member
    melgross said:

    apple isn’t working that way. Much of what Siri does now is done one the device itself. This is why Apple has been working on the on-chip AI so hard. Apple’s method doesn’t rely on massive amounts of data, particularly personal data. The AI is actually deeper than than that in deep eating. But, it’s a much harder technology to master. Because of that, Siri lags. But as we can see, Siri is improving faster. In another two or three years, it should begin to regain the lead. It’s been said that Apple’s ongoing persistence with privacy is holding Siri back, and it’s true. But Apple’s work is far less dependent on personal data, and if they can pull this off, then they’ll have a big advantage.
    By definition AI is using a lot of data.  This is practically impossible to do this on the device only, except for a limited number of cases.  If Apple insists on doing most of the Siri locally on the device, it will only get further behind.

    The privacy is a bogus argument.  Customers would not complain if Apple would say:  "If you as a customer give your explicit consent, we will be collecting your personal information to improve the answers Siri is giving to you. This gathered information will not be used for something else".  In fact a lot of customers are convinced this is already the case.

    Apple remains a device selling company.  Siri is not a strategic product for Apple, it exists to support the hardware offering.  As long as customers don't go to the competition because of the perceived weakness of Siri, nothing will change in this respect.  Bear in mind that in non-English languages the gap with Google is even bigger.



  • Reply 9 of 19
    melgross said:
    Apple needs to improve the international functions of this and that of Maps.
    The first thing I do after reading "Apple needs to do this" is to stop reading.  Pointing out deficiencies is easy, anybody can do that.  What the "Apple needs to do" crowd fail to do is offer solutions.  They can't because they don't have any.  If they did they'd have their own development firm offering such a solution, or they'd be working for Apple.  They don't so their opinions are worthless.

    The truth is that products only get improved when they are hindering sales.  SIRI and Maps are not hindering sales.

    Could SIRI be better?  Of course, but so could Google's 85% accurate voice assistant.  The delta between 78.5% accuracy and 85% accuracy is 6.5 errors out of 100 attempts.  This is not a deal breaker when you consider the alternative: buying an Android smartphone with Google Voice vs buying an iPhone with SIRI.

    Will SIRI and Maps get better?  Undoubtedly, but improvements from 78.5% to 85% are hardly noticeable in the real world, and even less so as accuracy goes above 85%.
    StrangeDays
  • Reply 10 of 19
    melgross said:
    Much of what Siri does now is done on the device itself. This is why Apple has been working on the on-chip AI so hard. Apple’s method doesn’t rely on massive amounts of data, particularly personal data. The AI is actually deeper than than that in deep eating. But, it’s a much harder technology to master. Because of that, Siri lags. But as we can see, Siri is improving faster. In another two or three years, it should begin to regain the lead. It’s been said that Apple’s ongoing persistence with privacy is holding Siri back, and it’s true. But Apple’s work is far less dependent on personal data, and if they can pull this off, then they’ll have a big advantage.
    A beautiful observation as to the differences between SIRI and everyone else's voice assistant. I forget the name of the firm Apple bought about 2 years ago, but their machine learning technology required only 3% of the total samples that all others require, and was more accurate.   I believe 3 years is the outside time frame whereby SIRI surpasses everyone else in voice recognition accuracy.
  • Reply 11 of 19
    melgrossmelgross Posts: 33,510member
    melgross said:
    Much of what Siri does now is done on the device itself. This is why Apple has been working on the on-chip AI so hard. Apple’s method doesn’t rely on massive amounts of data, particularly personal data. The AI is actually deeper than than that in deep eating. But, it’s a much harder technology to master. Because of that, Siri lags. But as we can see, Siri is improving faster. In another two or three years, it should begin to regain the lead. It’s been said that Apple’s ongoing persistence with privacy is holding Siri back, and it’s true. But Apple’s work is far less dependent on personal data, and if they can pull this off, then they’ll have a big advantage.
    A beautiful observation as to the differences between SIRI and everyone else's voice assistant. I forget the name of the firm Apple bought about 2 years ago, but their machine learning technology required only 3% of the total samples that all others require, and was more accurate.   I believe 3 years is the outside time frame whereby SIRI surpasses everyone else in voice recognition accuracy.
    that’s the estimates I’ve seen.

    i notice that I didn’t catch the typo “deep eating”. Well, I guess it’s right, because it does eat a lot of data.
    edited August 2018 douglas bailey
  • Reply 12 of 19
    melgrossmelgross Posts: 33,510member

    melgross said:
    Apple needs to improve the international functions of this and that of Maps.
    The first thing I do after reading "Apple needs to do this" is to stop reading.  Pointing out deficiencies is easy, anybody can do that.  What the "Apple needs to do" crowd fail to do is offer solutions.  They can't because they don't have any.  If they did they'd have their own development firm offering such a solution, or they'd be working for Apple.  They don't so their opinions are worthless.

    The truth is that products only get improved when they are hindering sales.  SIRI and Maps are not hindering sales.

    Could SIRI be better?  Of course, but so could Google's 85% accurate voice assistant.  The delta between 78.5% accuracy and 85% accuracy is 6.5 errors out of 100 attempts.  This is not a deal breaker when you consider the alternative: buying an Android smartphone with Google Voice vs buying an iPhone with SIRI.

    Will SIRI and Maps get better?  Undoubtedly, but improvements from 78.5% to 85% are hardly noticeable in the real world, and even less so as accuracy goes above 85%.
    You don’t agree that Apple has to improve this for the almost 65% of their customers who aren’t in the USA?  So it’s ok if we get much better service from Apple’s software than anyone else? That’s a very atrange bit of thinking.

    usually I don’t say that Apple must, or should do anything. I just say that I think they should do it, which is different. But for this, yes, they need to do it if they’re going to bring the percentage of users up, which we know they want to do.

    its not a solution that I have to present here. I don’t even know why you would say that. All that needs to be done is to bring the software and service up to our level. Apple knows how to do this. It’s not a solution they need. They just have to hire the people needed. When they began to improve maps, they said they hired thousands of people to do it. But most of their efforts center here. They’ve stated that they want to have the best mapping service of anyone. They’ve said the same thing about Siri. Now, they just have to do it. No new solutions required other than effort and some of that cash pile. The difference between 85% and 78.5% is noticeable. Going from 78.5 to 85 is 8% and if you get 8 more wrong answers out of a hundred, you will notice it.
    edited August 2018
  • Reply 13 of 19
    GabyGaby Posts: 190member
    That’s great and all I just wish they would do something similar for Apple music/iTunes. I’m Italian and I listen to a lot of foreign particularly French, Spanish and Italian as well as more obscure world music and it’s a nightmare trying to use Siri in this regard. Would be nice if they at least added a way to teach it how to pronounce artists and songs (similar to teaching how to pronounce contacts) as a stopgap measure until they can remedy the issue completely. 
    douglas bailey
  • Reply 14 of 19
    melgrossmelgross Posts: 33,510member
    Gaby said:
    That’s great and all I just wish they would do something similar for Apple music/iTunes. I’m Italian and I listen to a lot of foreign particularly French, Spanish and Italian as well as more obscure world music and it’s a nightmare trying to use Siri in this regard. Would be nice if they at least added a way to teach it how to pronounce artists and songs (similar to teaching how to pronounce contacts) as a stopgap measure until they can remedy the issue completely. 
    I agree.
    douglas baileyGaby
  • Reply 15 of 19
    melgross said:
    Much of what Siri does now is done on the device itself. 
    So you did keep reading.  :D


  • Reply 16 of 19
    the variations in US English language pales in comparison to the variations in UK English, good luck Apple when they decide to tackle that one. :D :D
  • Reply 17 of 19
    volcanvolcan Posts: 1,799member
    In the past one of my biggest complaints about Siri was her inability to understand Spanish names of places. Living in Southern California that is a major concern since so many of our streets, locations and businesses have Spanish names. I can now say that Siri is greatly improved in this regard even when the request is spoken using proper Spanish enunciation. Before you had to Spanglish it to have any hope of her understanding what you were asking. Her pronunciation of Spanish names has improved as well. She still needs some more work in that area but it is better than it was.
    edited August 2018
  • Reply 18 of 19
    melgrossmelgross Posts: 33,510member
    I’m wondering how this will work for all of the people who are in one area, but come from somewhere else. If it’s assumed that people will speek a certain way in each area, will it be worse for people who don’t speak that way in an area? Will recognition be worse that before?

    how does Apple manage this? People don’t stay in one place.
  • Reply 19 of 19
    melgross said:
    I’m wondering how this will work for all of the people who are in one area, but come from somewhere else. If it’s assumed that people will speek a certain way in each area, will it be worse for people who don’t speak that way in an area? Will recognition be worse that before?

    how does Apple manage this? People don’t stay in one place.
    Hmm, right. Like, if I have a heavy Boston accent and go to LA and ask Siri where the closest “bah” is. I know that’s not a perfect example but I tink I’m follawin ya. 
Sign In or Register to comment.