AppleInsider › Forums › Mobile › iPhone › Street test measures Siri comprehension at 83%, accuracy at 62%
New Posts  All Forums:Forum Nav:

Street test measures Siri comprehension at 83%, accuracy at 62% - Page 4

post #121 of 141
6/30/12 at 12:16pm
Quote:
Originally Posted by Hellacool View Post

…you know the ones you are ignoring.  on top of the fact that most consumers have zero idea what beta means.

I'll address them below. Also, I covered consumer intelligence.
Quote:
Keep grasping straws, it is entertaining.

Keep up the veiled insults, it's a good explanation.

You'll learn for what later. lol.gif
Quote:
Originally Posted by Blitz1 View Post

…a beta is never intended for a large audience. It's meant for a review group sending back findings on using the software.

I suppose you don't consider "iPhone 4S owners" to be a 'review group'.
Quote:
- does Siri beta have such reporting functions?… …Never can I inform Apple why the 38% failed and what I should have had.

Yes. You inform through using it. Any time you correct something it got wrong, you're informing Apple. Any time you enunciate further, you're informing Apple. They said this at launch.
Quote:
- was it launched as a feature for the whole of the intended audience from day 1?

Nope.
Quote:
- does it implicate that when Siri is out of beta, that it will be flawless (as current errors are so vividly attributed to a beta status?

No software is flawless, so there's no implication of anything.
Quote:
Originally Posted by JeffDM View Post

The part I am disappointed with is the only place the beta is noted is on the web site. The TV ads I've seen certainly don't say it's beta, and I hadn't seen it noted on the store displays. It just seems wrong to advertise it as your flagship feature and not make it clear in all your marketing materials that it's beta.

Agreed. I'm surprised the commercials are allowed to not say "beta". They say "sequences shortened", but nothing about the beta. I'd think the British whatever-it-is would be all over that (unless they DO do it there…).
PhilBoogie
That's Google alright. For a stupid company they sure do dumb things.
Reply
PhilBoogie
That's Google alright. For a stupid company they sure do dumb things.
Reply
post #122 of 141
6/30/12 at 8:46pm
Quote:
Originally Posted by JeffDM View Post

The part I am disappointed with is the only place the beta is noted is on the web site. The TV ads I've seen certainly don't say it's beta, and I hadn't seen it noted on the store displays. It just seems wrong to advertise it as your flagship feature and not make it clear in all your marketing materials that it's beta.

 

And under what law or sense of morality would that have to fall under?

 

According to the University of Ohio, for example, Apple's designation is perfectly correct. 

 

 

Quote:
A beta version or beta release software usually is the first version of a computer program that implements all features in the initial software requirements specification and is considered an intermediate stage. It can be unstable but useful. Generally considered a preview stage where testers (sometimes, the general public) are encouraged to use the beta software and provide feedback for the developers.

 

Apple has decided, and rightfully so, to label Siri as beta when and where they want to. Considering that Siri will always be in a 'learning' mode, calling it a beta, is a work of genius and is no different than Michelangelo often declaring, "I am still learning," even past his 90th birthday, as well, "It's no accident that services such as Gmail, Google Maps, Flickr, del.icio.us, and the like may be expected to bear a 'Beta' logo for years at a time."

 

As a matter of fact, Apple has every right to label Siri, Perpetual or Permanent Beta. Google that phrase and checkout the 62 million results.

 

 

 

 

Reply
Reply
post #123 of 141
6/30/12 at 9:43pm
Quote:
Originally Posted by Onhka View Post

And under what law or sense of morality would that have to fall under?

I didn't say anything about legality. Maybe you're responding to someone else on that point.

I think you misread what I said. I don't have any problem with it being called a beta or that it's made available as part of a shipping product. The problem I have is not making it clear in more of the marketing materials. The fact that Siri is beta isn't even in the small print on the ads that I saw.
Reply
Reply
post #124 of 141
7/1/12 at 4:14am
Quote:
Originally Posted by Tallest Skil View Post

They say "sequences shortened", but nothing about the beta. I'd think the British whatever-it-is would be all over that (unless they DO do it there…).

Well Australia is quite picky about this kind of stuff. Besides fining Apple for iPad "4G", if they wanted to they could go after Siri. The Australian consumer watchdog may have their hands full though policing companies increasing prices unfairly because of the carbon pricing just starting today in Australia.
Reply
Reply
post #125 of 141
7/1/12 at 4:16am
Quote:
Originally Posted by JeffDM View Post

I didn't say anything about legality. Maybe you're responding to someone else on that point.
I think you misread what I said. I don't have any problem with it being called a beta or that it's made available as part of a shipping product. The problem I have is not making it clear in more of the marketing materials. The fact that Siri is beta isn't even in the small print on the ads that I saw.

It is naughty of Apple. Somebody should make Apple state Siri [Beta] more clearly, otherwise Apple should just remove the Beta status altogether. It's not like an average consumer will be like, "Oh, it's Beta, that's okay then" ~ because outside of our cloistered circles the man/woman on the street has no idea what Beta is, let alone "artificial intelligence".

Apple needs to come off the fence on Siri, "double-down" like Tim said (as they are doing) even further, and go for broke with Siri. There's no turning back now, no need for "Beta"...
Reply
Reply
post #126 of 141
7/1/12 at 8:09am
Quote:
Originally Posted by sr2012 View Post

Well Australia is quite picky about this kind of stuff. Besides fining Apple for iPad "4G", if they wanted to they could go after Siri.

I'm quite surprised they haven't already. Maybe Siri works well enough with Australian accents that no one has complained about it.
PhilBoogie
That's Google alright. For a stupid company they sure do dumb things.
Reply
PhilBoogie
That's Google alright. For a stupid company they sure do dumb things.
Reply
post #127 of 141
7/1/12 at 10:58am
Quote:
Originally Posted by Gatorguy View Post

I've no doubt that Mr. Munster will do a side-by-side test between Siri and Google Now once it's in release, as will hundreds of others. I don't expect miracles but would not be at all surprised that Google would offer the more accurate results between the two. I'd personally expect comprehension to be a draw. Both iOS and Android have pretty good voice recognition already.

By the way, isn't Munster generally a pro-Apple guy? Certainly no sign that he cares much for Google AFAIK. I'd be surprised if his tests were intended to make Siri look bad. 

The problem is with munsters argument is that he is compairing direct text input to an ai on a computer for accuracy. Its not a fair comparison. The text input comes from a human being, the other is a computer ai that is doing astoundingly well from recognizing voice input and then analyzing what you mean or want. Siri will get more accurate with time. It already is.
Reply
Reply
post #128 of 141
7/1/12 at 7:00pm
This is in response to Mechanic's statement: "The problem is with munsters argument is that he is compairing direct text input to an ai on a computer for accuracy. Its not a fair comparison. The text input comes from a human being, the other is a computer ai that is doing astoundingly well from recognizing voice input and then analyzing what you mean or want. Siri will get more accurate with time. It already is."

So text input comes from a human but voice input from a human into siri comes from an AI? Your comment makes absolutely no sense. Siri has been proven unreliable, but your apple colored glasses refuse to allow you to see the truth.
Reply
Reply
post #129 of 141
7/1/12 at 7:12pm
Quote:
Originally Posted by youngliz View Post

This is in response to Mechanic's statement:

We've a dedicated quote feature that lets you do this more elegantly.
Quote:
So text input comes from a human but voice input from a human into siri comes from an AI? Your comment makes absolutely no sense.

He's noting the additional steps required to make Siri function and how that makes the comparison illegitimate.
Quote:
…your apple colored glasses…

Yeah, you're gonna get along well here. 😕
PhilBoogie
That's Google alright. For a stupid company they sure do dumb things.
Reply
PhilBoogie
That's Google alright. For a stupid company they sure do dumb things.
Reply
post #130 of 141
7/1/12 at 8:37pm
Quote:
Originally Posted by Tallest Skil View Post

I'm quite surprised they haven't already. Maybe Siri works well enough with Australian accents that no one has complained about it.

Yeah, either that or people here aren't using Siri as much. I have to stick to "British English" as that works best for me, because my accent is not hardcore "aussie" enough and still somewhat British... Well, the way I was taught it, Queen's English and all that LOL.

But yeah. No major rumblings in the mainstream media about Siri. They're probably too busy showing our Treasurer holding up a piece of chicken at a supermarket to prove that there are no cost increases due to the new carbon pricing (tax).
Reply
Reply
post #131 of 141
7/2/12 at 5:03am
Quote:
Originally Posted by JeffDM View Post


I didn't say anything about legality. Maybe you're responding to someone else on that point.

 

You are right on both counts. Your point, I guess, was more towards the "morality".

 

 

Quote:
Originally Posted by JeffDM View Post
I think you misread what I said. I don't have any problem with it being called a beta or that it's made available as part of a shipping product. The problem I have is not making it clear in more of the marketing materials. The fact that Siri is beta isn't even in the small print on the ads that I saw.
 

 

First question, what ads are you referring to?

 

Are they the same type of ads that Google had for over 6 years for Google Maps, which most people didn't realize that they were beta. In fact now on the home site of their latest version, i.e., Google Maps Navigation for Android, there is no mention that it is Beta and one would be hard pressed to see it, even and, until you go onto the Google Play site.

 

My point is that for Siri to work at all, it has to learn. And will never stop learning to work well. It will never be perfect. But I suggest that it is more perfect than Googling is right now. And until someone examines the millions of results it gets for virtually every query and determines that it is more accurate I will continue to stand by my position. Imagine one taking a exam and giving literally millions of answers each time. Think they will pass.

 

Try, "How do I get to New York city?" Siri gives one correct answer. Google gives you over 3 and a half million, but not one comes close to being correct.

Reply
Reply
post #132 of 141
7/2/12 at 5:08am
Quote:
Originally Posted by Onhka View Post

 

 

 

 

First question, what ads are you referring to?

 

Are they the same type of ads that Google had for over 6 years for Google Maps, which most people didn't realize that they were beta. In fact now on the home site of their latest version, i.e., Google Maps Navigation for Android, there is no mention that it is Beta and one would be hard pressed to see it, even and, until you go onto the Google Play site.

I've never seen a TV ad for Google Maps. When did you see one?

melior diabolus quem scies

"No theatrics and no more personal attacks, just stick to the logic and tell me why I don't have any argument ~ Jragosta/2012

Reply

melior diabolus quem scies

"No theatrics and no more personal attacks, just stick to the logic and tell me why I don't have any argument ~ Jragosta/2012

Reply
post #133 of 141
7/2/12 at 6:19am
Quote:
Originally Posted by Onhka View Post

First question, what ads are you referring to?

The TV ads for Siri. I have no problem with Apple promoting beta software or services (this is the second reiteration of that point that I've made), but in order to argue "it's beta" as a defense, I think they could easily do a better job telling people up front that it's beta.
Reply
Reply
post #134 of 141
7/2/12 at 7:36am
Sadly, Gene Munster found that Siri "... is two years behind Google." So, a service designed primarily as a personal assistant which is only eight months old is already nearly as good when used as a search engine as a competing service primarily designed as a search engine which is fifteen years old.

Gene Munster also found that "Google replies accurately 86% of the time."

Seems likely to me that many of the inaccuracies and inadequacies of Siri as a search engine are the result of Google Search.

I find it amazing that anyone would continue to use a search engine with only 86% accurate results. Great news that Apple continues to develop Siri and displace Google from more results.
Edited by MacBook Pro - 7/2/12 at 7:44am
Reply
Reply
post #135 of 141
7/2/12 at 8:03am
why does the accent thing always comes up in this articles about siri.
It is obviously not an issue if it is rated b+ on comprehension, it is actually a short coming of the software. other software has a training in setup that usually gets around any accent issues. after that they are usually fairly accurate.
the main problem was due to siri understanding what was being asked, then after looking it up online getting the wrong answer.
Steve Woz had the same criticism..
I consider this argument to be racist when people actually don't know how the person speaks and alludes it to be the problem.
people all over the country have different accents from New York to Boston Texas and Oregon.
Reply
Reply
post #136 of 141
7/2/12 at 8:10am
Quote:
Originally Posted by missingxtension View Post

I consider this argument to be racist when people actually don't know how the person speaks and alludes it to be the problem.
people all over the country have different accents from New York to Boston Texas and Oregon.

??? We're talking about say different accents in the US, UK, Australia, etc. for English alone... That's no small feat for advanced speech recognition with very, very little per-user training at the start. What argument is racist?
Reply
Reply
post #137 of 141
7/2/12 at 9:24am
Quote:
Originally Posted by missingxtension View Post

I consider this argument to be racist when people actually don't know how the person speaks and alludes it to be the problem.

What the heck does accent have to do with race? I consider racism racism when it isn't racism in the slightest.
PhilBoogie
That's Google alright. For a stupid company they sure do dumb things.
Reply
PhilBoogie
That's Google alright. For a stupid company they sure do dumb things.
Reply
post #138 of 141
7/2/12 at 9:47am
Quote:
Originally Posted by missingxtension View Post

why does the accent thing always comes up in this articles about siri.
It is obviously not an issue if it is rated b+ on comprehension, it is actually a short coming of the software. other software has a training in setup that usually gets around any accent issues. after that they are usually fairly accurate.
the main problem was due to siri understanding what was being asked, then after looking it up online getting the wrong answer.
Steve Woz had the same criticism..
I consider this argument to be racist when people actually don't know how the person speaks and alludes it to be the problem.
people all over the country have different accents from New York to Boston Texas and Oregon.


Since you mentioned Steve Wozniak, here is what he said:

'What are the five largest lakes in California?' and it came up one, two, three four five -- shocked me," Wozniak said, according to the Albany (N.Y.) Times Union. "And then I said, 'What are the prime numbers greater than 87?' and it came up starting with 91. That's pretty incredible."

"I'd say, 'What are the five largest lakes in California?' and it would have all these lakefront properties selling," Wozniak said. "And I'd say 'What are the prime numbers greater than 87?' and [Siri would answer], like, 'prime rib.'" ...

Siri has never been an advertising platform like Google Search. If Siri provided such answers then such is the result of Siri integration with Google Search. Incidentally, upon asking Siri these same questions, the answers I received were correct and concise.

Siri now does far, far more as a key component of iOS 6 "Sundance" than as an independent app developed by SRI.
Reply
Reply
post #139 of 141
7/2/12 at 10:29am
Quote:
Originally Posted by Gatorguy View Post

I've never seen a TV ad for Google Maps. When did you see one?

 

Can't say that I have either. I was referring to JeffDM' comment,

 

Quote:
Quote:
The fact that Siri is beta isn't even in the small print on the ads that I saw.

which from my learned experience having spent most of my working life, in marketing, advertising and education, has nothing to do with TV ads. If anything, Disclosure would be a better choice.

 

 

Quote:

Originally Posted by JeffDM View Post

The TV ads for Siri. I have no problem with Apple promoting beta software or services (this is the second reiteration of that point that I've made), but in order to argue "it's beta" as a defense, I think they could easily do a better job telling people up front that it's beta.

 

You have to be kidding. First of all most wouldn't know what it meant or even cared that it's beta.

 

And if they were to, imaging the effort and costs to explain it.

 

Quote:

Beta

Beta (named after the second letter of the Greek alphabet) is the software development phase following alpha. It generally begins when the software is feature complete. Software in the beta phase will generally have many more bugs in it than completed software, as well as speed/performance issues. The focus of beta testing is reducing impacts to users, often incorporating usability testing. The process of delivering a beta version to the users is called beta release and this is typically the first time that the software is available outside of the organization that developed it.

The users of a beta version are called beta testers. They are usually customers or prospective customers of the organization that develops the software, willing to test the software without charge, often receiving the final software free of charge or for a reduced price. Beta version software is often useful for demonstrations and previews within an organization and to prospective customers. Some developers refer to this stage as a preview, prototype, technical preview (TP), or early access. Some software is kept inperpetual beta—where new features and functionality are continually added to the software without establishing a firm "final" release.

 

Perhaps you should ask the same thing of Google.

 

Google Maps was in beta for 6 years. And now, Google Maps with Navigation. More telling, Googles's GMail, which is virtually in perpetual beta.

 

Interesting that the two Google Chrome TV ads highlight GMail. The current ad is all about GMail via Google Chrome. Much like Apple's iPhone 4S/ Siri ad. Neither has a beta logo or explanation in site. 


 

To a better understanding.

 

Quote:
Originally Posted by missingxtension View Post

why does the accent thing always comes up in this articles about siri.
It is obviously not an issue if it is rated b+ on comprehension, it is actually a short coming of the software. other software has a training in setup that usually gets around any accent issues. after that they are usually fairly accurate.
the main problem was due to siri understanding what was being asked, then after looking it up online getting the wrong answer.
Steve Woz had the same criticism..
I consider this argument to be racist when people actually don't know how the person speaks and alludes it to be the problem.
people all over the country have different accents from New York to Boston Texas and Oregon.

 

Perhaps a little reading is in order

 

Quote:

Does Siri work out of the box, or do I have to teach it?

Siri works right out of the box, without any work on your part. And the more you use Siri, the better it will understand you. It does this by learning about your accent and other characteristics of your voice. Siri uses voice recognition algorithms to categorize your voice into one of the dialects or accents it understands. As more people use Siri and it’s exposed to more variations of a language, its overall recognition of dialects and accents will continue to improve, and Siri will work even better.

Siri also uses information from your contacts, music library, calendars, and reminders to better understand what you say. So it responds more accurately when you ask to make a phone call, play music, or create an appointment or reminder.

 

Like I have iterated previously, calling Siri, Beta, is not illegal, morally wrong or misleading. In fact, by all definitions, it can never be final. Just like the human brain, it needs to continue to be fed or it dies. Unless one is dumber than a door knob.

 

But understanding that it would be permanent or perpetual beta

 

Quote:

Beta

Some software is kept in perpetual beta—where new features and functionality are continually added to the software without establishing a firm "final" release.

 

Much like Wikipedia has been described from an academic librarian, e.g.,

 

Quote:
Wikipedia is always a "permanent beta", always under constant revision.

 

Quote:
In a world where content and the software that houses it is constantly changing, a publication is never "done" and thus many effective publications exist (and indeed thrive) in a permanent state of beta.

 

Or Harold Jarche's Life in perpetuel Beta

 

Quote:
Perpetual beta is my attitude toward learning – I’ll never get to the final release and my learning will never stabilise. I’ve also realised that clients with a similar attitude are much easier to work with than those who believe that we will reach some future point where everything stabilises and we don’t need to learn or do anything else. I believe that this point is called death.

Edited by Onhka - 7/2/12 at 10:35am
Reply
Reply
post #140 of 141
7/2/12 at 9:16pm
Quote:

 

Originally Posted by Onhka View Post

 

 

Like I have iterated previously, calling Siri, Beta, is not illegal, morally wrong or misleading. In fact, by all definitions, it can never be final. Just like the human brain, it needs to continue to be fed or it dies. Unless one is dumber than a door knob.

 

But understanding that it would be permanent or perpetual beta

 

 

 

But a product always improving thanks to owner feedback doesn't define it as beta, else the Nest thermostat would be, or could call itself that, and they don't.   It's if the hardware/firmware/software is considered unfinished and not ready for release, not if it continues to improve or reach for better performance as the years go by.   There's hardly any technology, hardware or software, that that wouldn't qualify as beta with that kind of definition, and there's plenty of "learning" technologies that aren't sold as beta based on the fact that they will get better.  But now that Google and Apple have leaned on it it will enter the mainstream release strategy, no doubt, and half of what we use will be same as it ever was but labeled beta, and they'll be kicking themselves for not doing it ten years ago.

 

 

 

Quote:

Originally Posted by Onhka View Post

 

You have to be kidding. First of all most wouldn't know what it meant or even cared that it's beta.

 

And if they were to, imaging the effort and costs to explain it.

 

 

Not that this applies here, but when a company needs to rectify something, that the efforts and costs are staggering can be secondary to the fact that something needed rectifying.   

Reply
Reply
post #141 of 141
7/12/12 at 3:34am
Quote:
Originally Posted by MacBook Pro View Post

Since you mentioned Steve Wozniak, here is what he said:
'What are the five largest lakes in California?' and it came up one, two, three four five -- shocked me," Wozniak said, according to the Albany (N.Y.) Times Union. "And then I said, 'What are the prime numbers greater than 87?' and it came up starting with 91. That's pretty incredible."
"I'd say, 'What are the five largest lakes in California?' and it would have all these lakefront properties selling," Wozniak said. "And I'd say 'What are the prime numbers greater than 87?' and [Siri would answer], like, 'prime rib.'" ...
Siri has never been an advertising platform like Google Search. If Siri provided such answers then such is the result of Siri integration with Google Search. Incidentally, upon asking Siri these same questions, the answers I received were correct and concise.
Siri now does far, far more as a key component of iOS 6 "Sundance" than as an independent app developed by SRI.

I've noticed Siri gradually improving. The lakes and prime numbers question works fine. "Show me John Cusack" movies works.

I've still got some irritation because I use UK English recognition as that is what my accent matches with the most closely.

But this Siri thing, particularly with Mandarin and Cantonese recognition, is here to stay, and will only get better.

My only critique is Steve would have waited a year or so, it feels like Tim Cook is gunning things a little.
Reply
Reply
New Posts  All Forums:Forum Nav:
  Return Home
  Back to Forum: iPhone
AppleInsider › Forums › Mobile › iPhone › Street test measures Siri comprehension at 83%, accuracy at 62%