Apple researchers work to stop AI from taking actions you didn't approve
AI agents are learning to tap through your iPhone on your behalf, but Apple researchers want them to know when to pause.
Apple continues to refine AI agent capabilities
A recent paper from Apple and the University of Washington explored this disparity. Their research focused on training AI to understand the consequences of its actions on a smartphone.
Artificial intelligence agents are getting better at handling everyday tasks. These systems can navigate apps, fill in forms, make purchases or change settings. They can often do this without needing our direct input.
Autonomous actions will be part of the upcoming Big Siri Upgrade that may appear in 2026. Apple showed its idea of where it wants Siri to go during the WWDC 2024 keynote.
The company wants Siri to perform tasks on your behalf, such as ordering tickets for an event online. That kind of automation sounds convenient.
But it also raises a serious question: what happens if an AI clicks "Delete Account" instead of "Log Out?"
Understanding the stakes of mobile UI automation
Mobile devices are personal. They hold our banking apps, health records, photos and private messages.
An AI agent acting on our behalf needs to know which actions are harmless and which could have lasting or risky consequences. People need systems that know when to stop and ask for confirmation.
Most AI research focused on getting agents to work at all, such as recognizing buttons, navigating screens, and following instructions. But less attention has gone to what those actions mean for the user after they are taken.
Not all actions carry the same level of risk. Tapping "Refresh Feed" is low risk. Tapping "Transfer Funds" is high risk.
Building a map of risky and safe actions
The studystarted with workshops involving experts in AI safety and user interface design. They wanted to create a "taxonomy" or structured list of the different kinds of impacts a UI action can have.
The team looked at questions like -- Can the agent's action be undone? Does it affect only the user or others? Does it change privacy settings or cost money?
The paper shows how the researchers built a way to label any mobile app action along multiple dimensions. For example, deleting a message might be reversible in two minutes but not after. Sending money is usually irreversible without help.
The taxonomy is important because it gives AI a framework to reason about human intentions. It's a checklist of what could go wrong, or why an action might need extra confirmation.
Training AI to see the difference
The researchers gathered real-world examples by asking participants to record them in a simulated mobile environment.
Modeling the impacts of UI operations on mobile interfaces. Image credit: Apple
Instead of easy, low-stakes tasks like browsing or searching, they focused on high-stakes actions. Examples included changing account passwords, sending messages, or updating payment details.
The team combined the new data with existing datasets that mostly covered safe, routine interactions. They then annotated all of it using their taxonomy.
Finally, they tested five large language models, including versions of OpenAI's GPT-4. The research team wanted to see if these models could predict the impact level of an action or classify its properties.
Adding the taxonomy to the AI's prompts helped, improving accuracy at judging when an action was risky. But even the best-performing AI model -- GPT-4 Multimodal -- only got it right around 58% of the time.
Why AI safety for mobile apps is hard
The study found that AI models often overestimated risk. They would flag harmless actions as high risk, like clearing an empty calculator history.
That kind of cautious bias might seem safer. However, it can make AI assistants annoying or unhelpful if they constantly ask for confirmation when it is not needed.
The web interface for participants to generate UI action traces with impact. Image credit: Apple
More worryingly (and unsurprisingly), the models struggled with nuanced judgments. They found it hard to decide when something was reversible or how it might affect another person.
Users want automation that is helpful and safe. An AI agent that deletes an account without asking can be a disaster. An agent that refuses to change the volume without permission can be useless.
What comes next for safer AI assistants
The researchers argue their taxonomy can help design better AI policies. For example, users could set their own preferences about when they want to be asked for approval.
The approach supports transparency and customization. It helps AI designers identify where current models fail, especially when handling real-world, high-stakes tasks.
Mobile UI automation will grow as AI becomes more integrated into our daily lives. Research shows that teaching AI to see buttons is not enough.
It must also understand the human meaning behind the click. And that's a tall task for artificial intelligence.
Human behavior is messy and context-dependent. Pretending that a machine can resolve that complexity without error is wishful thinking at best, negligence at worst.
Read on AppleInsider
Comments
https://www.youtube.com/watch?v=HipTO_7mUOw&list=WL&index=36
Try telling a court, 'Sorry, my phone's AI bought a 1st class flight to Hawaii when I really wanted a bus ticket from NYC to Philly"
AI, as it is currently implemented, is an over hyped POS. Sure, there are a few things it can do for us, but there is no killer app. I don't want it anywhere near my devices.