Apple provides detailed reasoning behind abandoning iPhone CSAM detection
A child safety group pushed Apple on why the announced CSAM detection feature was abandoned, and the company has given its most detailed response yet as to why it backed off its plans.
Apple's scrapped CSAM detection tool
Child Sexual Abuse Material is an ongoing severe concern Apple attempted to address with on-device and iCloud detection tools. These controversial tools were ultimately abandoned in December 2022, leaving more controversy in its wake.
A child safety group known as Heat Initiative told Apple it would organize a campaign against the choice to abandon CSAM detection, hoping to force it to offer such tools. Apple responded in detail, and Wired was sent the response and detailed its contents in a report.
The response focuses on consumer safety and privacy as Apple's reason to pivot to a feature set called Communication Safety. Trying to find a way to access information that is normally encrypted goes against Apple's wider privacy and security stance -- a position that continues to upset world powers.
"Child sexual abuse material is abhorrent and we are committed to breaking the chain of coercion and influence that makes children susceptible to it," wrote Erik Neuenschwander, Apple's director of user privacy and child safety.
"Scanning every user's privately stored iCloud data would create new threat vectors for data thieves to find and exploit," Neuenschwander continued. "It would also inject the potential for a slippery slope of unintended consequences. Scanning for one type of content, for instance, opens the door for bulk surveillance and could create a desire to search other encrypted messaging systems across content types."
"We decided to not proceed with the proposal for a hybrid client-server approach to CSAM detection for iCloud Photos from a few years ago," he finished. "We concluded it was not practically possible to implement without ultimately imperiling the security and privacy of our users."
Neuenschwander was responding to a request by Sarah Gardner, leader of the Heat Initiative. Gardner was asking why Apple backed down on the on-device CSAM identification program.
"We firmly believe that the solution you unveiled not only positioned Apple as a global leader in user privacy but also promised to eradicate millions of child sexual abuse images and videos from iCloud," Gardner wrote. "I am a part of a developing initiative involving concerned child safety experts and advocates who intend to engage with you and your company, Apple, on your continued delay in implementing critical technology."
"Child sexual abuse is a difficult issue that no one wants to talk about, which is why it gets silenced and left behind, added Gardner. "We are here to make sure that doesn't happen."
Rather than take an approach that would violate user trust and make Apple a middleman for processing reports, the company wants to help direct victims to resources and law enforcement. Developer APIs that apps like Discord can use will help educate users and direct them to report offenders.
Apple doesn't "scan" user photos stored on device or in iCloud in any way. The Communication Safety feature can be enabled for children's accounts but won't notify parents when nudity is detected in chats.
The feature is being expanded to adults in iOS 17, which will enable users to filter out unwanted nude photos from iMessage. Apple hopes to expand these features to more areas in the future.
Read on AppleInsider
Comments
seriously, the world is awash with people who in the name of good, do great harm. This is a gold medal example.
and good on Apple for realising that many peoples’ concerns with how this system could create a situation that could be exploited by bad actors, and further, adopted those concerns as its own position.
Google is even worse for using machine learning to try and identify images that aren't in the CSAM database, generating additional false positives.
The UK politicians behind all of this are arrogant and stupid and refuse to listen to experts in the field.
Pragmatically, letting the unfortunate still enjoy the convenience of iMessage is better than dumping them to Green.
Pretty amazing they listened and turned around.
It's that simple.
I like that Apple would pull out - in western countries this can actually work.
(And the many users protested against the initial plan!)
I've posted this before but it's instructional when it comes to how bad "slippery slope" arguments are. During the original Edward Snowden hysteria about government surveillance powers, Yahoo! decided to sue the government over FISA subpoenas for their metadata. Yahoo!'s argument was that the FISA subpoenas would lead to abuse of power by the government. When the judge asked Yahoo! to provide examples of the government abusing FISA subpoena power they didn't have any. All they had was conjectural scenarios based on "slippery slope" arguments. Result? The case was thrown out of court.
There have been stories of families being contacted about possible sexual crimes because they posted facebook photos of their very young children innocently playing naked in a home swimming pool. My personal experience in the work environment - a medical clinic - was once the IT Dept implemented a "sexual content" filter/reporting system, when I or my colleagues were doing work research we'd trigger an alert that required a visit to HR. When looking up information about a dermatologist, the website might feature cosmetic surgery with an example of breast adjustments. The image would trigger the filter/alert.
Years ago, a local childcare facility had to close down because of accusations of sexual abuse. Turned out, one child was upset they didn't get their way about something and made up the story to "get back" at the care center. Once one report was made, parents of the other kids started questioning them - I'm guessing there was a bit of a FOMO frenzy. Soon there were many reports. But then, once there was time to do an actual investigation, one by one the accusations were withdrawn. Eventually, the facility was cleared of all wrongdoing. By then it was too late. Newspapers print "leading news" in big bold print. They print "oops - sorry" in small print where space will allow.
So if you are going to have a system, especially an "automatic" system for detecting bad action, you should have a system just as robust - or more so - for corrections when the "evil detector" gets it wrong. Because that can ruin lives.
No local scanning for non-iCloud-stored photos.
Sounds like you don’t understand how it works (intended to work) either.