As far as I can tell, consumer-facing AI isn't improving in leaps and bounds anymore, and probably hasn't for about a year or 18 months.
A couple weeks before Super Bowl I asked half a dozen LLM-powered AI assitant chatbots when the Super Bowl kickoff was scheduled. Not a single chatbot got it right.
Earlier today I asked several chatbots to fill out a 2025 NCAA mens basketball tournament bracket. They all failed miserably. Not a single chatbot could even identify the four #1 seeds. Only Houston was identified as a #1 seed by more than one chatbot, probably because of their performance in the 2024.
I think Grok filled out a fictitious bracket with zero upsets. There has never been any sort of major athletic tournament that didn't have at least one upset. And yet Grok is too stupid to understand this. It's just a dumb probability calculator that uses way too much electricity.
Context, situational awareness, common sense, good taste, humility. Those are all things that AI engineers have not programmed yet into consumer facing LLMs.
An AI assistant really need to be accurate 99.8% of the time (or possibly more) to be useful and trustworthy. Getting one of the four #1 seeds correct (published on multiple websites) is appallingly poor. If it can't even identify the 68 actual teams involved in the competition, what good is an AI assistant? Why would you trust it to do anything else? Something more important like schedule an oil change for your car? Keep your medical information private?
As I said a year ago, all consumer facing AI is still alpha software. It is nowhere close to being ready for primetime. In several cases there appears to be some serious regression.
25% right isn't good enough. Neither is 80%. If a human assistant failed 3 out of 4 tasks and you told them so, they would be embarrassed and probably afraid that they would be fired. And yes, I would fire them.
Apple senior management is probably coming to grips with this. If they put out an AI-powered Siri that frequently bungles requests, that's no better than the feeble Siri they have now. And worse, it'll probably erodes customer trust.
"Fake it until you make it" is not a valid business model. That's something Elizabeth Holmes would do. And she's in prison.
Did you try Gemini, currently 2.0 Flash? In a voice search on my Pixel it listed South Auburn Tigers, West Gators, East Duke, and Midwest Cougars
I did not give Gemini the bracket question. I did give it the Super Bowl question which it failed like the others.
Your comment brings up an important illustrative point. No one has the time to dork around with 7-8 AI chatbots to find one (or more) that gives the correct answer for each question. That's not a sustainable approach.
There's probably some AI chatbot that will might get the right answer to a simple question. The problem is no AI chatbot is reliably accurate enough to instill trust and confidence. I can't ask ten questions to 8 chatbots and wade through the responses. In the same way, having ten human personal assistants isn't a worthwhile approach.
Let's say Grok has a 20% accuracy score and Gemini is 40%. That's double the accuracy for Gemini but it still is way too low to be trusted and deemed reliable.
FWIW, Gemini has made major moves in the past 60 days, so it might be worth giving them more of a shot. But yeah, point make about reliability. 80% isn't good enough to be trusted. That's still solidly in trust but verify territory.
Like I said earlier Gemini also failed to answer the simple question "When is the Super Bowl kickoff?" less than two weeks before the actual game.
Gemini is untrustworthy. All AI ASSistants are untrustworthy.
80% is still way too low. I would fire any human personal assistant if that was their success rate. Your standards sucks.
Your reading for comprehension apparently sucks. You might read my post again, that one where I agreed with you that 80% isn't good enough.
That said, wouldn't it be great if each of us here could be right at least 80% of the time? Instead, we all have those members we generally trust but still verify, the news articles we generally trust but still verify, and our time-tested research sources that we generally trust but still verify.
Besides your partner, is there any source you have that's "right" (which itself needs defining) 100% of the time?
Totally not surprised this happened although I didn't expect it to take this long. Tim Cook is the proverbial iron fist in a velvet glove. It's worth recounting this tale from ancharlesn said:
Totally not surprised this happened although I didn't expect it to take this long. Tim Cook is the proverbial iron fist in a velvet glove. It's worth recounting this tale from an urgent meeting Cook had called:
"One day back then, he convened a meeting with his team, and the discussion turned to a particular problem in Asia. “This is really bad,” Cook told the group. “Someone should be in China driving this.” Thirty minutes into that meeting Cook looked at Sabih Khan, a key operations executive, and abruptly asked, without a trace of emotion, “Why are you still here?”
Khan, who remains one of Cook’s top lieutenants to this day, immediately stood up, drove to San Francisco International Airport, and, without a change of clothes, booked a flight to China with no return date, according to people familiar with the episode. The story is vintage Cook: demanding and unemotional."
urgent meeting Cook had called:
"One day back then, he convened a meeting with his team, and the discussion turned to a particular problem in Asia. “This is really bad,” Cook told the group. “Someone should be in China driving this.” Thirty minutes into that meeting Cook looked at Sabih Khan, a key operations executive, and abruptly asked, without a trace of emotion, “Why are you still here?”
Khan, who remains one of Cook’s top lieutenants to this day, immediately stood up, drove to San Francisco International Airport, and, without a change of clothes, booked a flight to China with no return date, according to people familiar with the episode. The story is vintage Cook: demanding and unemotional."
But siri has been a diaster for 10yrs now and I remember alexa came out and created a big buzz then Google suddenly Google made a big effort to take on alexa but appl sat there and did nothing. The only reason he is acting now is coz the stock is getting crushed but appl has a much bigger AI problem then siri. AI is not only gonna effect phones but in 5yrs 40% of the white color jobs will disappear. This AI john G they hired from Google claiming he was head of AI at Google is bullshit coz Google has so many AI ppl this guy was I think was doing some search with AI which he tried to all to iPhones but beyond that he is pretty useless
So now the lead of one failed product is taking over another failed product? What could possibly go wrong?
Vision Pro has been a complete failure for Apple. Overpriced. No developers. No one cares about AR. It doesn't solve any problem. Walk into an Apple Store and no one pays attention to them. For decades, no one has cared about AR because they don't want goggles on their head. Just like no one wanted to wear 3D glasses to watch TV and 3D TVs are non-existent. So why would anyone spend $3500 for a heavy pair of goggles on their head?
And now this guy is going to fix Siri? The same Siri that Apple has pretty much abandoned for the past 14 years? Gets things wrong constantly and it is inconsistent on the Mac, iPhone, AppleTV and the speaker. Good luck with that.
Apple Intelligence has been a huge disappointment so far. Siri still makes constant mistakes on simple dictation for a text message. Siri, do you want me to send this? NO!
I think all tech companies r dillusional about VR with metaverse and now even Samsung and Google r getting back into it. I use VR everyday for last 8yrs and it has no market
But AR is different I think that might be the future now since we have so much AI
The thing is, Siri cannot even tell you what month it is. Try it, “Siri, what month is it”.
I thought you may have been exaggerating when you wrote this, but I asked Siri just now what month it is, and it replied with “I don’t understand”. Unreal.
Those of us who upgraded to the iPhone 16 Pro based upon what was promised, but never delivered should receive an apology from Apple and even compensation,
There have long been rumours about differing opinions between him and others at Apple on AI/Siri.
It's possible that those differences were linked to foundational problems and that he preferred a clean sweep approach and there was pushback. Who knows!
My guess is that Apple just didn't have the manpower to get its projects to port on time. Some were left to sink with bits salvaged (Titan maybe) and others hit stormy seas along the way.
If he has been moved off his project it's possible that a few months from now, when things settle, he'll announce he's leaving and then move on.
I'm sure we'll hear more about what went on at Apple around the time ChatGPT hit the market and the all hands meetings that happened. I'm convinced that Apple (for whatever reason) just didn't see all this coming so soon.
However it's also important to point out that Siri is not a P&L center unlike Apple TV+ or iCloud or Fitness+ or the Apple Watch hardware division.
Technically true, but,,, how does an all-in Apple guy like myself end up with Alexa and 3 Echo Dots in my home? Simple: Alexa has always worked much better as a voice assistant than Siri, plus there's an entire world of products with voice-control capability that support Alexa but don't support Siri. The most glaring for me is probably my Sonos system, but I have a long list of others in my home. So in the ongoing battle for control of the home, I don't know how Apple wins without a more capable and reliable Siri.
Is he on temporary leave or just straight up out of there? (As in completely fired)
Probably strolling the halls. Well, not exactly strolling the halls, but walking around in circles at Apple Park.
As an aside if you are the boss - read: Mr Tim - you demote a subordinate - read: John - and clear the decks and air. You don't commit Hara Kiri and fire yourself.
The thing is, Siri cannot even tell you what month it is. Try it, “Siri, what month is it”.
I thought you may have been exaggerating when you wrote this, but I asked Siri just now what month it is, and it replied with “I don’t understand”. Unreal.
Totally not surprised this happened although I didn't expect it to take this long. Tim Cook is the proverbial iron fist in a velvet glove. It's worth recounting this tale from an urgent meeting Cook had called:
"One day back then, he convened a meeting with his team, and the discussion turned to a particular problem in Asia. “This is really bad,” Cook told the group. “Someone should be in China driving this.” Thirty minutes into that meeting Cook looked at Sabih Khan, a key operations executive, and abruptly asked, without a trace of emotion, “Why are you still here?”
Khan, who remains one of Cook’s top lieutenants to this day, immediately stood up, drove to San Francisco International Airport, and, without a change of clothes, booked a flight to China with no return date, according to people familiar with the episode. The story is vintage Cook: demanding and unemotional."
What he should have done was look up at the ceiling and yelled “KHAAAN!”.
Tim’s ai, headset, and car haven’t panned out so far.
=
Would love to see a return to the days of Apple not releasing something until it’s 100% ready and captivating. I’ll give Cook’s car a pass since it was never officially announced.
Even the watch was closely guarded. And it was a success when it finally launched. A return to form in that regard would be most welcome.
No more public experiments and betas please.
I'll never understand what people mean when they say this other than "I don't like it so it didn't pan out." By what metrics are you measuring the panning?
Other than delaying one feature, Apple Intelligence is doing fine. Apple didn't promise a sentient machine like others in the space did. Is that a bad thing? Them delaying something that isn't ready instead of releasing it anyway is exactly what you're asking Apple to do, yet you're criticizing them for it.
And unless you know something about how Apple Vision Pro has done so far and Apple's expectation for the product, there's no way of knowing "how it panned out." It's been pretty awesome from my perspective.
I’ll never understand why some people have to put words in other people’s mouths to pretend to have a point.
Failure has nothing to do with opinion. Even a product considered good by popular opinion can fail in the marketplace. It’s a known quantity that headsets are niche. Apple was up against a wall to begin with going in this direction. Sure, you personally may be super into the headset thing. But most people aren’t. And most people did not buy one. It’s just not a hit. But you can pretend it is doing better because “I just love it so much!” Or you can pretend others objectively see it as a flop because “feelings.” It’s neat. And that’s about it. It’s not essential, nor does it fit with anyone’s lifestyle but the most sedentary among us.
No one said apple promised a sentient machine. But their AI JUST ISNT COMPETITIVE AT THE MOMENT. Samsung phones do a far better job of ai photo editing for one example. Siri is horrible. It’s ok for basic, trivial things. Beyond that, it gets frustrating. Again, the competition is better. This being Apple, I still believe they will come out on top, especially considering the focus and investment, not to mention ethical training. But as-is, it’s a bit of a flop.
So yes, as a shipping product, criticism is well deserved. We aren’t looking at the future. We are looking at what apple has produced and is working right now, live, on our devices. I’ve not criticized apple for delays. Wrong again. I’ve only advocated they do so more instead of shipping half-baked stuff like everybody else does. Hence the desire to see the company return to secretive development and only releasing something when it’s ready to be the best in its class. Basically the opposite of what you wrote.
Yes, we all know your perspective in the headset. That’s not what matters when assessing its floppiness. A quick search yields all kinds of info on how well it’s done. It ain’t good. Maybe it would be considered successful if your neighbor launched a startup and made it. But not for a company like Apple. The Zune sold over 3 times as much and was considered a failure by Microsoft, who buried it. Sure, it was a different product category that saw iPod selling in the gazillions, but 1.2 million sales in a year is nothing to sneeze at… unless you’re Microsoft or Apple, in which case, it’s not viewed as a success. Especially when your CEO ignores staff concerns and gushes all over the media about it - only for the shipping product to receive a tepid reception.
That’s just the reality at the moment.
The light at the end of the tunnel is that apple doesn’t give up so easily and will likely distill the tech into a pair of Oakley’s. By then their AI should be mature and ready to shine on such a lifestyle device.
I don't need LLM-AI at the moment, just as I did not need a smart phone till early 2010s. So there is hope that Apple will catch on and eventually continue to lead the AI consumption for the common people like me (not tech sophisticated but I think a bulk of apple customers are in my category)- what was disheartening so far was the neglect of Siri ( esp understanding different accents and commands) given how long it has been around. Hopefully the big shots at apple are coming to their senses. iPhone is what Apple has - rest of all other things exist because of the ecosystem that revolves around it. They cannot sleep at the wheel any longer
Tim’s ai, headset, and car haven’t panned out so far.
Would love to see a return to the days of Apple not releasing something until it’s 100% ready and captivating. I’ll give Cook’s car a pass since it was never officially announced.
Even the watch was closely guarded. And it was a success when it finally launched. A return to form in that regard would be most welcome.
No more public experiments and betas please.
I'll never understand what people mean when they say this other than "I don't like it so it didn't pan out." By what metrics are you measuring the panning?
Other than delaying one feature, Apple Intelligence is doing fine. Apple didn't promise a sentient machine like others in the space did. Is that a bad thing? Them delaying something that isn't ready instead of releasing it anyway is exactly what you're asking Apple to do, yet you're criticizing them for it.
And unless you know something about how Apple Vision Pro has done so far and Apple's expectation for the product, there's no way of knowing "how it panned out." It's been pretty awesome from my perspective.
The Vision Pro doesn’t sell. The only reason Cook gets away with it is that Apple has several cash cow products that bring in ungodly gobs of money. If the first iMac in 1998 sold as poorly as the Vision Pro Apple would probably be a nostalgia brand today like Atari.
Tim’s ai, headset, and car haven’t panned out so far.
Would love to see a return to the days of Apple not releasing something until it’s 100% ready and captivating. I’ll give Cook’s car a pass since it was never officially announced.
Even the watch was closely guarded. And it was a success when it finally launched. A return to form in that regard would be most welcome.
No more public experiments and betas please.
I'll never understand what people mean when they say this other than "I don't like it so it didn't pan out." By what metrics are you measuring the panning?
Other than delaying one feature, Apple Intelligence is doing fine. Apple didn't promise a sentient machine like others in the space did. Is that a bad thing? Them delaying something that isn't ready instead of releasing it anyway is exactly what you're asking Apple to do, yet you're criticizing them for it.
And unless you know something about how Apple Vision Pro has done so far and Apple's expectation for the product, there's no way of knowing "how it panned out." It's been pretty awesome from my perspective.
The Vision Pro doesn’t sell. The only reason Cook gets away with it is that Apple has several cash cow products that bring in ungodly gobs of money. If the first iMac in 1998 sold as poorly as the Vision Pro Apple would probably be a nostalgia brand today like Atari.
To be fair to the Vision Pro, it was never expected to be a mass market hit with version 1. It’s a niche and expensive product aimed at starting a new direction in computing.
The iMac, by comparison, was keenly priced and aimed at creating a new beachhead in an established market,
Tim’s ai, headset, and car haven’t panned out so far.
=
Would love to see a return to the days of Apple not releasing something until it’s 100% ready and captivating. I’ll give Cook’s car a pass since it was never officially announced.
Even the watch was closely guarded. And it was a success when it finally launched. A return to form in that regard would be most welcome.
No more public experiments and betas please.
I'll never understand what people mean when they say this other than "I don't like it so it didn't pan out." By what metrics are you measuring the panning?
Other than delaying one feature, Apple Intelligence is doing fine. Apple didn't promise a sentient machine like others in the space did. Is that a bad thing? Them delaying something that isn't ready instead of releasing it anyway is exactly what you're asking Apple to do, yet you're criticizing them for it.
And unless you know something about how Apple Vision Pro has done so far and Apple's expectation for the product, there's no way of knowing "how it panned out." It's been pretty awesome from my perspective.
I’ll never understand why some people have to put words in other people’s mouths to pretend to have a point.
Failure has nothing to do with opinion. Even a product considered good by popular opinion can fail in the marketplace. It’s a known quantity that headsets are niche. Apple was up against a wall to begin with going in this direction. Sure, you personally may be super into the headset thing. But most people aren’t. And most people did not buy one. It’s just not a hit. But you can pretend it is doing better because “I just love it so much!” Or you can pretend others objectively see it as a flop because “feelings.” It’s neat. And that’s about it. It’s not essential, nor does it fit with anyone’s lifestyle but the most sedentary among us.
No one said apple promised a sentient machine. But their AI JUST ISNT COMPETITIVE AT THE MOMENT. Samsung phones do a far better job of ai photo editing for one example. Siri is horrible. It’s ok for basic, trivial things. Beyond that, it gets frustrating. Again, the competition is better. This being Apple, I still believe they will come out on top, especially considering the focus and investment, not to mention ethical training. But as-is, it’s a bit of a flop.
So yes, as a shipping product, criticism is well deserved. We aren’t looking at the future. We are looking at what apple has produced and is working right now, live, on our devices. I’ve not criticized apple for delays. Wrong again. I’ve only advocated they do so more instead of shipping half-baked stuff like everybody else does. Hence the desire to see the company return to secretive development and only releasing something when it’s ready to be the best in its class. Basically the opposite of what you wrote.
Yes, we all know your perspective in the headset. That’s not what matters when assessing its floppiness. A quick search yields all kinds of info on how well it’s done. It ain’t good. Maybe it would be considered successful if your neighbor launched a startup and made it. But not for a company like Apple. The Zune sold over 3 times as much and was considered a failure by Microsoft, who buried it. Sure, it was a different product category that saw iPod selling in the gazillions, but 1.2 million sales in a year is nothing to sneeze at… unless you’re Microsoft or Apple, in which case, it’s not viewed as a success. Especially when your CEO ignores staff concerns and gushes all over the media about it - only for the shipping product to receive a tepid reception.
That’s just the reality at the moment.
The light at the end of the tunnel is that apple doesn’t give up so easily and will likely distill the tech into a pair of Oakley’s. By then their AI should be mature and ready to shine on such a lifestyle device.
All of this is moot because you're ignoring the core of my argument: you don't have any idea what Apple's success metric is for Apple Vision Pro, so there's no way to say if it is successful or not. You can move the goalposts and have an opinion on how successful it is, but your opinion isn't relevant. All anyone knows is it sold, content continues to be released for it, and a next-generation device is due in the next year.
As far as Apple Intelligence goes, it's still moot. I understand you have strongly held opinions on it, as many do. You incorrectly lump Siri in with Apple Intelligence, which is a separate feature. Yes, Siri needs work, but that doesn't mean it is useless either. I control my home, music, reminders, and timers just fine with Siri. I do wish it could do more, and I hope future updates and Apple Intelligence help with that. I'm interested in seeing what an LLM Siri looks like.
Everything Apple announced for Apple Intelligence works as advertised. One feature that was announced didn't launch and was delayed into later in 2025. We're still within the window of the expected release of the new contextual features.
And I wouldn't call what Samsung does "photo editing" as much as ruining a perfectly good photo.
You're welcome to your opinion. Just as always, don't mix them up with facts. We can disagree on things, but realize that we don't know what we don't know. It doesn't matter how well other companies are doing. What did Apple want to accomplish, and did it? You and I can't know that.
The thing is, Siri cannot even tell you what month it is. Try it, “Siri, what month is it”.
It’s an absolute embarrassment to Apple, has been since its release and yet they’ve not bothered to fix it.
Yes, we’ve had promises and the occasional enhancement. But after 14 years on the market it’s still a useless pile of shit and will likely to continue to be.
That query gave me the exact date “Saturday 5th of April, 2025”. (I’m behind in my reading).
Comments
You might read my post again, that one where I agreed with you that 80% isn't good enough.
That said, wouldn't it be great if each of us here could be right at least 80% of the time? Instead, we all have those members we generally trust but still verify, the news articles we generally trust but still verify, and our time-tested research sources that we generally trust but still verify.
Besides your partner, is there any source you have that's "right" (which itself needs defining) 100% of the time?
But AR is different I think that might be the future now since we have so much AI
It's possible that those differences were linked to foundational problems and that he preferred a clean sweep approach and there was pushback. Who knows!
My guess is that Apple just didn't have the manpower to get its projects to port on time. Some were left to sink with bits salvaged (Titan maybe) and others hit stormy seas along the way.
If he has been moved off his project it's possible that a few months from now, when things settle, he'll announce he's leaving and then move on.
I'm sure we'll hear more about what went on at Apple around the time ChatGPT hit the market and the all hands meetings that happened. I'm convinced that Apple (for whatever reason) just didn't see all this coming so soon.
As an aside if you are the boss - read: Mr Tim - you demote a subordinate - read: John - and clear the decks and air. You don't commit Hara Kiri and fire yourself.
Hopefully the big shots at apple are coming to their senses. iPhone is what Apple has - rest of all other things exist because of the ecosystem that revolves around it. They cannot sleep at the wheel any longer
The iMac, by comparison, was keenly priced and aimed at creating a new beachhead in an established market,
Everything Apple announced for Apple Intelligence works as advertised. One feature that was announced didn't launch and was delayed into later in 2025. We're still within the window of the expected release of the new contextual features.
And I wouldn't call what Samsung does "photo editing" as much as ruining a perfectly good photo.
You're welcome to your opinion. Just as always, don't mix them up with facts. We can disagree on things, but realize that we don't know what we don't know. It doesn't matter how well other companies are doing. What did Apple want to accomplish, and did it? You and I can't know that.