Merk182

UNTIL, of course, a host nation - say - the Chinese government - passes a LAW stating in order to "protect the privacy of its citizens"; organizations using any technology to scan photos (such as what Apple proposed) must compare to a database hosted on servers that physically reside within the host country borders, and this database content must be vetted by said government, to "prevent false accusations"  - then - if an issue is found through AI, the human review team, which also must be housed within the host country borders, must be conducted under the supervision of Chinese Government Officials, all this again, to "protect" any potential false accusations. This "Government Vetted" Database could then include hashes of identifying groups they wish ill upon, opposition partiies, or whatever, then a "hit" on their phone will give the host government all the reason it needs to put these "good guys" in prison for a very long time. 

Apple cannot say, we do not have the technical capability - because they Apple just created it. The back end is is just a search and comparing hashes with a database of known data - The Chinese Government could easily demand that for iPhones sold / registered for use in China Apple must to point to a different database for comparison. The Chinese Government could then manipulate the database to identify whatever they want and the human reviewers could very well be either 1) employees of the Chinese GOV who support the efforts, or 2) fearful of saying anything because of possible retaliation. 

It doesn't have to be China - insert "Putin" - "Lukashenko" of Belrus, or any dictator you care -

As it is now - Apple doesn't provide that capability - because it doesn't exist, but, once the capability is created - it can be abused, doesn't need to be hacked. Just controlled.  For all that Apple says that it will refuse - it really only "refuses" when it is the FBI demanding they do something, they often make adjustments favoring the Chinese Government, otherwise, they'd get kicked out of the country. In these cases such as, blocking free speech, or apps that enable free communication outside Chinese controlled channels, Apple bows to the demands of the government in the name of "obeying local law of the land. Apple cannot even drop it in a future iOS release if it starts to be abused, because this corrupt government can just say, any updated iOS Version that doesn't support this kind of search is not secure and cannot be rolled out in their country until it does. Nothing Apple can do about it. 

So, that what happens when local law of the land is corrupt and can easily be turned to bad outcomes. 

So - no matter the noble purposes behind the effort - a lot of really, really bad things can come out of it. 

Apple should NOT move forward.