Apple Intelligence summaries are still screwing up headlines

AppleInsider

The UK's BBC is continuing to complain about the notification summaries created by Apple Intelligence, with iPhone users being misinformed by misinterpreted headlines.

Examples of notification summaries on an iPhone

In December, the BBC complained to Apple about the summarization features of Apple Intelligence. While meant to save time for users, the summaries sometimes get things quite wrong, as it did with news headlines from the UK broadcaster.

In its new complaint on January 3, the BBC has been informed of summaries appearing on iPhones that were very inaccurate.

One summary, based on a headline from Thursday evening, claimed that darts player Luke Littler had "won PDC World Championship." In reality, Littler had only completed a semi-final, with the final itself to be held on Friday night.

In a second example, headlines from the BBC Sport app were condensed down with the claim that "Brazilian tennis player Rafael Nadal comes out as gay." It is believed that the real headline was for a story about Brazilian gay tennis player Joao Lucas Reis da Silva, and not the Spanish Nadal.

"As the most trusted news media organisation in the world, it is crucial that audiences can trust any information or journalism published in our name and that includes notifications," urged a BBC spokesperson. "It is essential that Apple fixes this problem urgently - as this has happened multiple times."

The BBC is not the only organization complaining about the lack of accuracy from Apple Intelligence when it comes to headlines. Reporters Without Borders said in December that it was "very concerned by the risks posed to media outlets" by the summaries.

The issues proved to the group that "generative AI services are still too immature to produce reliable information for the public."

Apple has yet to publicly comment on the current complaints, but CEO Tim Cook did acknowledge that inaccuracies would be a potential issue in June. While the results would be of "very high quality," the CEO admitted it may be "short of 100%."

In August, it was reported that Apple had preloaded instructions to counter hallucinations, instances when an AI model can create information on its own. Phrases "Do not hallucinate. Do not make up factual information" and others are fed into the system when acting on prompts.

Fixing the accuracy can also be tough for Apple, since it does not actively monitor what users are seeing on devices, as part of its policies on user privacy. The priority of using on-device processing where possible also makes it harder for Apple to work the problem.



Read on AppleInsider

daalseth

AI summaries and news. What could possibly go wrong. /s
People will look back in years to come and laugh at how dumb we must have been to trust an AI to tell us the important stuff. Honestly this will only end when people start getting killed from this misinformation and companies start getting sued. 

alex1n

This just prompted me to go looking in Settings turn all summaries off. It was getting too annoying anyway, regardless of accuracy. Whew.

lotones

It's only been out a month BBC. Have a nice warm cuppa get-over-yourself, and maybe try to focus on something else until Apple fixes it.

nubus

lotones said:

It's only been out a month BBC. Have a nice warm cuppa get-over-yourself, and maybe try to focus on something else until Apple fixes it.

Apple misuses the BBC logo and brand with headlines faking everything from suicides to sports results. Any company would be furious about being so haphazardly used as a guinea pig. Apple is liable for these messages just as it is for guiding users to cross rivers on bridges that don't exist. There is a reason why Tesla FSD doesn't drive.

When BBC makes a mistake it is on BBC. Did anyone ask for Apple to make a fake news generator while using corporate logos?

fred257

Apple should be held legally liable for this. Apple Intelligence? Just another way to sell the same iPhone they have had for years already. 

applezulu

nubus said:

lotones said:

It's only been out a month BBC. Have a nice warm cuppa get-over-yourself, and maybe try to focus on something else until Apple fixes it.

Apple misuses the BBC logo and brand with headlines faking everything from suicides to sports results. Any company would be furious about being so haphazardly used as a guinea pig. Apple is liable for these messages just as it is for guiding users to cross rivers on bridges that don't exist. There is a reason why Tesla FSD doesn't drive.

When BBC makes a mistake it is on BBC. Did anyone ask for Apple to make a fake news generator while using corporate logos?

If nothing else, Apple should more prominently remind users that these AI features are still in beta testing mode. As I recall, each user had to ask to be ‘on a list’ to receive access to AI. This was a formality, as access was immediately granted, but I think the purpose of that process is to define the feature as being in beta. 

With that in mind, the comparison to erroneous GPS prompts is apt. While there is a reasonable expectation that maps should be kept up to date and accurate, if you drive into a river where there is no bridge, a significant portion of the blame is on you for listening to the prompts while disregarding the reality there in front of you. 

The BBC is certainly justified in its objections here. Still, any reader who believes any headline, whether generated by humans or AI, without reading the content below it fails the most basic of media literacy tests. Particularly in this internet age, headlines are designed to draw attention far more than they are to communicate an accurate summary with poetic brevity. In fact, the fundamental problem for these AI summaries is in the high expectations built into its intent to do the latter, rather than the former. 

chasm

Turn off summaries = good suggestion for users

Hire a person or three to write unbiased, accurate summaries based on them actually reading the stories = better solution for Apple (we expect this type of crap from Google)

Someday AI will be able to do accurate summaries. But if today isn't that day, don't use AI for a job it clearly can't yet do.

edited January 4

dewme

Apple Intelligence is still Jell-O in liquid form.

It will harden ... eventually.

But it will still be Jell-O.

Don't sweat the small stuff.

dewme

The Apple Weather app on my iPhone reports that I received 126 inches of snow yesterday and 201 inches of snow today. I’d say it was closer to 8-10 inches total over two days. It’s only off by a little over 26 feet, which is a very good thing. 

marvin

chasm said:

Hire a person or three to write unbiased, accurate summaries based on them actually reading the stories = better solution for Apple (we expect this type of crap from Google)

Someday AI will be able to do accurate summaries. But if today isn't that day, don't use AI for a job it clearly can't yet do.

The volume of info is likely too high for human summaries for every news service in every country, every single day and they can't read summaries of personal messages for privacy. There could be a separate AI that checks the summaries for errors. This AI would compare the summary and the main text to see if there's a mismatch. At the very least, they should flag sensitive summaries that mention things like people dying, for human review. When personal messages get flagged for either sensitive content or the summary doesn't make sense, it can fall back to cropped text.

A news summary like "Brazilian tennis player Rafael Nadal comes out as gay." would get flagged as sensitive and raised for human review. Until the review is done, it can show an important sentence from the article cropped to fit the space.

carmb

As a retired journalist, I would like to point out that humans themselves often get it wrong. I lost count of how many times I was dismayed by a misleading headline attached to a piece I had written or worse, sloppy editing that had entirely misrepresented what I believed to be the truth and had attempted in my original copy to convey. Whenever I attended a press conference it was fun afterwards to note that in reading the various articles based on it, you would think we had all attended separate sessions. Without question we need to guard against simply relinquishing control to artificial devices even more disconnected from reality than us flawed humans can be. It is, however, naive of us to have thought getting this right would be an easy task.