Street test measures Siri comprehension at 83%, accuracy at 62%

123468

Comments

  • Reply 101 of 143
    hellacoolhellacool Posts: 759member

    Quote:

    Originally Posted by Tallest Skil View Post





    Saying Apple doesn't indicate it's a beta is flat out lying, too.

    216

    246

    I really shouldn't have to link to something as simple as Apple.com to debunk the things you say. Doing so also proves that a much larger number of people than you assume know it's a beta.




    Sigh, I know it is hot out and your brain may be running slow but I am referencing the televised commercials, the commercials majority see, the ones with the celebs, the ones Apple pours all its advertising dollars into, you know the ones you are ignoring.  on top of the fact that most consumers have zero idea what beta means.  Keep grasping straws, it is entertaining.

  • Reply 102 of 143
    hellacoolhellacool Posts: 759member

    Quote:

    Originally Posted by Apple ][ View Post


     


    Nah, people like me don't end up in jail.





    That's what all the guilty say.

  • Reply 103 of 143
    blitz1blitz1 Posts: 438member

    Quote:

    Originally Posted by Tallest Skil View Post





    Saying Apple doesn't indicate it's a beta is flat out lying, too.

    216

    246

    I really shouldn't have to link to something as simple as Apple.com to debunk the things you say. Doing so also proves that a much larger number of people than you assume know it's a beta.


    Well, as a former project manager, I can tell you this: a beta is never intended for a large audience.

    It's meant for a review group sending back findings on using the software.


    - does Siri beta have such reporting functions?


    - is it intended for a review group?


    - was it launched as a feature for the whole of the intended audience from day 1?


    - does it implicate that when Siri is out of beta, that it will be flawless (as current errors are so vividly attributed to a beta status?


     


    Now, for me, Siri works quite well. I do believe it's slightly better than the 62% and I can live with that.


    But please, don't BS me with a beta status. Never can I inform Apple why the 38% failed and what I should have had.

  • Reply 104 of 143
    hellacoolhellacool Posts: 759member

    Quote:

    Originally Posted by Blitz1 View Post


    Well, as a former project manager, I can tell you this: a beta is never intended for a large audience.

    It's meant for a review group sending back findings on using the software.


    - does Siri beta have such reporting functions?


    - is it intended for a review group?


    - was it launched as a feature for the whole of the intended audience from day 1?


    - does it implicate that when Siri is out of beta, that it will be flawless (as current errors are so vividly attributed to a beta status?


     


    Now, for me, Siri works quite well. I do believe it's slightly better than the 62% and I can live with that.


    But please, don't BS me with a beta status. Never can I inform Apple why the 38% failed and what I should have had.





    Shhhh, Shhhhh no logic or facts allowed here.

  • Reply 105 of 143
    gatorguygatorguy Posts: 24,213member

    Quote:

    Originally Posted by JerrySwitched26 View Post


     


     


     


    See what I mean?





    There's nothing good to be gained by intentionally trying to stir trouble. Not a great idea if you wish to stick around longer than last time. Way too many other members post trollish comments which add nothing to the discussion but trouble, I guess erroneously thinking they're being cute or creative. Instead they come off as juvenile and petty. Don't join them.

  • Reply 106 of 143
    imt1imt1 Posts: 87member

    Quote:

    Originally Posted by ankleskater View Post




    Such a comparison does not make sense. There's no way it would have been published in a peer-reviewed journal. A proper test would compare apples to apples, with control built in. If it makes sense to compare accuracy of speech processing to text input, why not include sign language and smoke signals too?


     


    This has nothing to do with the utility of Siri. The reported test just does not pass any meaningful scientific scrutiny.



     


    I don't mind the comparison. Its basically all about comparing the info you are asking Siri and the results vs. what you would get if you inputed the same info in a Google search box. I did not read the full study so I assume that it was the Exact same info with no additional help with the use of parameters in search. If the latter, its not a fair analysis. 


     


    Then there is another thing to consider. Currently, Apple has been using Google for the search. Question is are the different results because of Siri not transcribing accurately or is Google intentionally not passing the best results back to Siri? My guess is Google has to know where the searches are coming from and Google. Remember, No ads are passed, like a regular search engine so Google sees no benefit form providing results so who knows. Only way to know for sure is to compare the text output that Siri passes to Google. Something they do not see in the test. 


     


    As far as the Beta tag for Siri, I have no problem with it. I think its two fold. One is that Siri has a limited set of functionality/commands. Apple doesn't want those to feel its full featured. Second, is that Siri needed to build up its vocabulary and learn speech patterns and dialects from thousands of users to increase accuracy across the board. This also goes to point 1. How frustrating would it be to have a wider range of commands and features, which will be present in iOS6, then to get poor results and apps not launching. My guess is that if someone studied the current functions of Siri and the vocabulary it can respond to, you will probably find that those utilize required verbiage to quickly build out better speech recognition and nuances in speech.  I am also guessing that while Siri still might show Beta, the commercials may not mention it since Siri may have built up its speech recognition past the point of "Beta". Thus, issues in recognition for some after launch are no longer present today.  


    Thus, the online moniker would be removed to then coincide with the iOS6 release.    

  • Reply 107 of 143

    Quote:

    Originally Posted by jragosta View Post







    I have a PhD in science.


     


     


     


    This reminds me of the scene in Good Will Hunting where Ben Afflek is trying to pick up a Harvard girl by telling her that he thinks that they have a class together.


     


    "Oh?  Which class?"


     


    "History".


     


    LOL


     


    Me, personally, I've never heard of a PhD saying that they have a degree in a vague, general area. 

  • Reply 108 of 143
    dick applebaumdick applebaum Posts: 12,527member

    Quote:

    Originally Posted by MacBook Pro View Post





    My wife is born and raised Japanese, listening to her trying to coax information from Siri in English is hysterically funny. Siri in Japanese seems to be fairly reliable for her though.


     


    Interesting point.  Siri in iOS 6 has several dialects of most languages, e.g.  Spanish (Spain), Spanish (Mexico), Spanish (United States).


     


    Likely, Siri will evolve to understand mixed languages... Eso es Scheiße! (SpanDeutsche).

  • Reply 109 of 143
    imt1imt1 Posts: 87member

    Quote:

    Originally Posted by Blitz1 View Post


    Well, as a former project manager, I can tell you this: a beta is never intended for a large audience.

    It's meant for a review group sending back findings on using the software.


    - does Siri beta have such reporting functions?


    - is it intended for a review group?


    - was it launched as a feature for the whole of the intended audience from day 1?


    - does it implicate that when Siri is out of beta, that it will be flawless (as current errors are so vividly attributed to a beta status?


     


    Now, for me, Siri works quite well. I do believe it's slightly better than the 62% and I can live with that.


    But please, don't BS me with a beta status. Never can I inform Apple why the 38% failed and what I should have had.



    While I posted some possible reasons for calling it Beta, your post "A Beta is never intended for a large audience" needs clarification. What is a large audience. remember, while Siri when released it could handle some foreign languages. However, the functionality in those countries was limited as well. Thus, while the iPhone 4S was and is released in lots of countries, Siri can't work fully or at all in many. Google used to release everything as Beta and stayed that was for years. The audience was huge. Also, we are talking iPhone 4S users here which is a subset of a larger audience of total iPhone users with a significant number still on iPhone 4 and 3Gs. At the next launch, many if not most of those will be able to upgrade with their carrier to the new phone and thus most iPhone users will then be on a Siri enabled device. 

  • Reply 110 of 143
    dick applebaumdick applebaum Posts: 12,527member

    Quote:

    Originally Posted by mjtomlin View Post


     


    Like Gmail?


     


    Siri has only been out for 8 months... Gmail, an email service was out for years before it went off beta. Email was a tried and true technology for over 20 years at the time!!! Why in the hell was it ever in beta in the first place?


     


    Siri is a learning technology, meaning it needs to learn your voice and enunciation. Sorry do you not know what that means? It needs to learn how you talk, slurs and all.


     


    I think int he past 7 months since I've had my iPhone, Siri has dissed me a handful of times (maybe four or five).



     


    Exactly!   And Siri does learn!


     


    When I first started using Siri, she had trouble with family names...


     


    Especially my oldest grandson -- First name: Braden;  Last name Starts with "Gi".


     


    Siri pronounced these wrong -- she used a short 'a' in Braden instead of a long 'a';  and a 'Ji' instead of a hard 'G' sound for the last name.


     


    Over time, Siri learned the correct pronunciation by keeping track of my usage.


     


     


    Siri, What's my name?


     


    Siri:  "Your  Dick, but since we're friends, I get to call you 'Oh, Omnipotent One'".


     


     


    She had trouble with "Omnipotent" (don't they all), but eventually got it right.

  • Reply 111 of 143
    mstonemstone Posts: 11,510member

    Quote:

    Originally Posted by Dick Applebaum View Post


     


    When I first started using Siri, she had trouble with family names...


     


     



     


    You should hear her butcher some of my contact names.


     


    Xianli  has to be pronounced Zi-on-lee when actually phonetically it is only two syllables - jian-li


     


    Also my friend Venicio of course sounds like "B" not "V" so again you have to say it horribly wrong to make it work.


     


    When the Spanish version comes out you will be known as 'Dee-ca'

  • Reply 112 of 143


    Personally, I find that Google's accuracy and reliability when entering queries with text have been on a downhill slide for at least a year — without Siri being involved at all.  I wouldn't want to base a study of anything's reliability or accuracy when using Google.

  • Reply 113 of 143
    gatorguygatorguy Posts: 24,213member


    I guess I missed this until just now, but read at theVerge that voice recognition and dictation now works without a data connection on Google's latest OS, processing directly on your device. Fairly surprising. 

  • Reply 114 of 143
    hirohiro Posts: 2,663member

    Quote:

    Originally Posted by jragosta View Post





    Are you a flipping idiot or what?

    I have a PhD in science. I know what is scientific.

    Now, a scientific study may or may not have validity, but if it meets the premises of science, it is a scientific study. That is, it must have:

    1. A testable hypothesis. Check

    2. A methodology for testing the hypothesis. Check.

    3. A control. Check.

    4. Results. Check.

    5. A conclusion. Check.

    It meets the criteria. They are being careful to ensure that it's not being used as the be-all and end-all of research on the subject, but to say that it's not scientific just because it doesn't have a lot of replication is just plain wrong.

    And, no matter how you slice it, it's infinitely more valid than a Motorola advertisement saying that Motorola's phones are better than Apple's phones. That should be obvious to anyone thinking at above a 5 year old's level. Sadly, that seems to leave you out.


     


    Pretty sad statement for a PhD I think.  You are completely forgetting (or worse--ignoring) that 1, 2, 3 & 5 need to be properly executed and carefully crafted to remove bias from the measurements.  Without that your 1-5 just give us pseudo science, which is  worse than something that is merely unscientific.  Why worse?  Pseudo science is a lie cloaked in the trappings and lingo of science in an attempt to deceive.  


     


    I vote that Munster was unscientific. He wasn't attempting to lie, he just doesn't know how to craft an unbiased study.

  • Reply 115 of 143
    jragostajragosta Posts: 10,473member
    hiro wrote: »
    Pretty sad statement for a PhD I think.  You are completely forgetting (or worse--ignoring) that 1, 2, 3 & 5 need to be properly executed and carefully crafted to remove bias from the measurements.  Without that your 1-5 just give us pseudo science, which is  worse than something that is merely unscientific.  Why worse?  Pseudo science is a lie cloaked in the trappings and lingo of science in an attempt to deceive.  

    So tell us what was done with bias in the comparison I provided.
  • Reply 116 of 143
    macbook promacbook pro Posts: 1,605member
    imt1 wrote: »
    While I posted some possible reasons for calling it Beta, your post "A Beta is never intended for a large audience" needs clarification. What is a large audience. remember, while Siri when released it could handle some foreign languages. However, the functionality in those countries was limited as well. Thus, while the iPhone 4S was and is released in lots of countries, Siri can't work fully or at all in many. Google used to release everything as Beta and stayed that was for years. The audience was huge. Also, we are talking iPhone 4S users here which is a subset of a larger audience of total iPhone users with a significant number still on iPhone 4 and 3Gs. At the next launch, many if not most of those will be able to upgrade with their carrier to the new phone and thus most iPhone users will then be on a Siri enabled device. 

    His statement is false.

    There is not an industry standard for the audience of a beta.

    There isn't even an industry standard for the term "beta" (although "beta" is generally accepted to be a version for limited distribution which may have issues not generally acceptable to the public).

    There is a trend in the last decade for very large "public beta" releases of highly complicated software with complicated algorithms (artificial intelligence or computer graphics), constrained resources (to evaluate performance at load) or expansive environments (simulated "worlds"). Siri arguably meets all three criteria for a large public beta as listed.

    Some businesses have added gamma and omega releases.
  • Reply 117 of 143
    lenolenolenoleno Posts: 27member


    I don't understand why Siri will not accept written requests.


     


    Voice in a noisy environment will always be challenging.


     


    The value of Siri is not just voice.  Siri is smarter than Google cause it editorializes.


     


    For example "remind me when i leave"

  • Reply 118 of 143
    q2hq2h Posts: 18member


    Siri is in beta because the technology requires significant use to to develop and understand it's user base to eventually be as comprehensive and intuitive as possible.  All requests are sent to Apple's servers to be analyzed and processed. It's a system of learning and can't simply be mastered in a few revisions. This is a work in progress. If Siri wasn't in beta, people would lose their minds if it went down or didn't handle a request as the wished. I personally have no gripes to think of (I own a 4S) and am anticipating some welcome changes in iOS 6.


     


    If the argument is that you don't think Siri works as well as it does in the commercials where dramatizations speed up requests, then maybe you should be upset with the rest of the marketing world as well.  Do a fancy pair of basketball shoes make you better at basketball? Does an expensive liquor make you classy? Will a Happy Meal actually make you happy when you finally consume it?  More people than you'd think are walking around with their intellect off and respond to pure emotional stimulation.  It's an advertisers job to cater to that market since they're the mass majority. There will always more stupid people (less inquisitive and more dependent) than smart people and stupid people are an easy sale.  Smart people are more inquisitive and require more stimulation to their intellect, rather their senses and emotions. The "green" market is one example for the more intellectual -- which also helps Apples market share being that their products use highly recyclable components.  The Toyota Prius is another example for that market base.


     


    When intelligent people are forced to consume, they do research and use their knowledge to determine what is best for them.


    If you need all you information from a commercial and are mad at Apple for somehow misleading you with theirs, I know where you stand.


     


    As fas as tests go, I require specifics. I don't trust percentages.  For example, if you were to hear on the news or read poll results that say 80% of the voters are against legalizing marijuana, I'd want to know how many voters there were and what was their demography? Meanwhile, the polls may have been answered by elderly republicans or religious fanatics (more republicans, lol) that still think the movie Reefer Madness holds water and had no idea that the paper companies were behind the misleading propaganda to eliminate the competition (hemp).


     


    Off topic..


    How many haters actually use Apple products? I've met a few Apple haters in my time and they've all converted because their tired old rhetoric was just that.  They saw the light when their friends were using MBPs while they were replacing or repairing laptops too frequently and wasting money. Not to mention all the bloatware, spyware and other common malicious attacks that are inherent to nearly all Windows machines. Unfortunately, most are still forced to use Windows for work, but that works on the same hardware so there's no problem there.


     


    People commonly make the mistake of buying a PC thinking the higher spec'd hardware will outperform and Mac.  This is still true in some areas.  Apple uses quality hardware that run at a more comfortable pace.  PCs put a bunch of top performing hardware in a plastic shell.  Time after time the higher performing hardware will burn itself out.  Apple products are a more refined technology and manage performance in a more optimal and reliable manner.  Personally, I wouldn't buy any other machine unless, like others have mentions, they totally screw up big like Nokia and RIM, which seems unlikely being that Apple is the one putting them in their place.  My first computers were hand-me-down PCs. My first purchase was a Mac.


     


    I can still remember the day I was driving home with my new iPod classic in '04 and I said to my passenger "Now if they'd only make a phone out of this so I won't have to carry both."  Sure enough, Apple did and wow has the market changed because of them.

  • Reply 119 of 143
    onhkaonhka Posts: 1,025member


    Quote:

    Originally Posted by AppleInsider 



    While the convenience of Apple's Siri comes with searching by voice, a new test has found that Apple's personal assistant is not yet a reliable enough search engine to entirely replace text input with Google.

    Of the 83 percent of queries Siri could comprehend, 21 percent were answered inaccurately. Some of the questions that stumped Siri were:


    • "What team does Peyton Manning play for?" Siri would respond with the answer to the previous query.


    • "Where is Elvis buried?" Siri looked for a person named "Elvis Buried."


    • "Where am I?" Sometimes a map pin would be dropped in the wrong place.


    • "When did the movie Cinderella come out?" Siri would attempt to find current theater movie times via Yelp.


    • "How do I get from Boston to New York?" Siri responded: "I can only give directions from your current location. I can't give you directions to a place you are not in."



    In Munster's test, he gave Google an A+ for comprehension and B+ for accuracy, while Apple's Siri was graded with a B for comprehension and a D for accuracy.

    His testing also found that Siri relies on Google for 60 percent of its answers in iOS 5. But this fall, with the launch of iOS 6, Munster projects that Siri's reliance on Google will drop to 48 percent, replacing the search giant with Apple's own solutions for mapping and sports scores.

    "While Siri is two-plus years behind Google today, we are optimistic about its future," Munster wrote. "With the iOS 6 release in the fall, we expect Siri to improve meaningfully while reducing its reliance on Google."



     


    Interesting questions. But unless I haven't done my due diligence, most of the questions exampled above do not fall into the capabilities presented on the Apple Siri feature page, except for the links to the listed internet sites it queries. As such, perhaps the protocol, of which we have not be aprised, provides the answers that all qualified researchers understand, i.e., "Ask a stupid question and you get a stupid answer."


     


    Certainly a Google text-search should give more accurate answers than Siri is or every will be capable. But giving more accurate answers does not mean they are all accurate. For example Googling, "where is elvis buried" and you get over 6 million hits. And not all answers given, e.g., "Paul The Octopus 'To Record Elvis Tribute Album," mentions anything about where Elvis is buried.


     


    As for, "How do I get from Boston to New York?" Siris does exactly as it is supposed to do. Ask the exact question via Google and much of the 1.4 billion results aren't accurate. To be picky, my youngest grandson would have simply answered, "By walking, flying, car, bus, train, boat, swimming, sailing…." My older granddaughter would have been more guiding, but would not have outlined the "Getting From New York City to Boston" route first as Google did.


     


    When I showed this article to my wife, her response was immediate and had me Google three questions while she Siri'd her iPhone.


     


    1. When is our anniversary?


    2. When is my birthday?


    3. Where is your iPhone?


     


    Needless to say, Google couldn't answer correctly to any of the three as Siri couldn't on the third question. But as she pointed out, "Siri is beta. Just like most men will always be. By the way, your iPhone is where you left it, in the…and I suggest you use be nice to Siri in the next few weeks."


     

  • Reply 120 of 143
    jeffdmjeffdm Posts: 12,951member
    blitz1 wrote: »
    Well, as a former project manager, I can tell you this: a beta is never intended for a large audience.

    It's meant for a review group sending back findings on using the software.
    - does Siri beta have such reporting functions?
    - is it intended for a review group?
    - was it launched as a feature for the whole of the intended audience from day 1?
    - does it implicate that when Siri is out of beta, that it will be flawless (as current errors are so vividly attributed to a beta status?

    Now, for me, Siri works quite well. I do believe it's slightly better than the 62% and I can live with that.
    But please, don't BS me with a beta status. Never can I inform Apple why the 38% failed and what I should have had.

    I think you can blame Google. They rolled out "beta" products for the world to use and had that designation out there for years. I never noticed a bug submission button anywhere with those products.

    The part I am disappointed with is the only place the beta is noted is on the web site. The TV ads I've seen certainly don't say it's beta, and I hadn't seen it noted on the store displays. It just seems wrong to advertise it as your flagship feature and not make it clear in all your marketing materials that it's beta.
Sign In or Register to comment.