Remix.run Logo
avidiax 7 hours ago

> It is crazy people think apple isnt on the side of privacy.

> It also ensured pressure from governments and plaintiffs, including CSAM victims, who preferred Apple’s more interventionist approaches, which Apple had voluntarily demonstrated it was willing to do.

I feel that Apple open pandora's box with the client-side scanning. It proved that it was technically feasible, and was "privacy preserving". I use scare quotes there because I don't think that political or religious dissidents would find that the same or similar technology used to discover and persecute them is "privacy preserving". And that's really the problem with Apple here. They provided a model for scanning for any kind of message or material while purportedly maintaining privacy.

bayindirh 6 hours ago | parent | next [-]

> It proved that it was technically feasible, and was "privacy preserving".

Didn't their paper disproved by reversing the perceptual hashes to reveal blurred version of the images being hashed, and Apple basically said "that's fair, it's not as robust as we wanted, let's visit this later"?

If not, I'll happily stand corrected, but please share sources.

Addenda:

- Apple's original paper: https://web.archive.org/web/20210807165030/https://www.apple...

- Paper breaking the hash: https://arxiv.org/abs/2111.06628

Edit: The second one is the wrong paper. I’ll find and link the correct one tomorrow. Keeping the link for transparency.

comex 5 hours ago | parent | next [-]

The paper you linked doesn’t reveal blurred versions of the images being hashed. It does train a classifier to determine which of 1,000 ImageNet classes an image belongs to, which “achieved a top-1 test accuracy of 4.34%”.

bayindirh 4 hours ago | parent [-]

Then, that’s the wrong paper. I’ll find it and link it as a reply to this comment. Probably tomorrow morning.

yogorenapan 5 hours ago | parent | prev [-]

> to reveal blurred version of the images being hashed

Skimmed your linked paper. It seems they were able to classify hashes up to ~8% top-1 accuracy and ~30% top-10. Not exactly a blurred version, or any images at all.

So for example, they can say that you probably have images of trees, or images of buildings, but without much other data & very low accuracy.

I'd still be a lot more concerned about them simply flagging political images rather than trying to get a broad understanding of what type of photos I have

soulofmischief 5 hours ago | parent [-]

It starts at a broad understanding and ends with people permanently giving up their right and ability to keep rogue corporate governments in check.

GeekyBear 4 hours ago | parent | prev | next [-]

> I feel that Apple open pandora's box with the client-side scanning.

That box has been open for years now.

Big brother is already watching what you do on your Android device.

> A Dad Took Photos of His Naked Toddler for the Doctor. Google Flagged Him as a Criminal.

https://www.nytimes.com/2022/08/21/technology/google-surveil...

xboxnolifes 10 minutes ago | parent | next [-]

I'm pretty sure this article if from after Apple introduced (floated the idea of?) client-side scanning. I'm not sure it really bolsters the idea that it's been open for years.

letmevoteplease 4 hours ago | parent | prev | next [-]

This is not client side. Images just sitting on your Android phone are probably safe (although Google could push an update at any time). Your images get scanned when you back them up, send them over RCS, etc. I have even seen criminal cases originating from reverse image search - anything that touche the servers of the big tech companies, except apparently Apple, will be scanned using questionable AI and against a secret list to Protect the Children.

GeekyBear 3 hours ago | parent [-]

This is an image that he did not send off of his device to anyone except his doctor's office, yet Google reached into his private data and scanned it anyway.

Google reported him to the police based on a single false positive.

To add insult to injury, even after the police contacted Google to tell them that they had cleared him of wrongdoing, Google refused to restore access to his account.

izacus 3 hours ago | parent [-]

Stop making stuff up man, the image was uploaded to Google Photos servers.

> The father uploaded photos of his son’s genitals, which were also backed up on his Google cloud, to the health care provider’s messaging system as requested.

GeekyBear 3 hours ago | parent | next [-]

Did he set up his device to upload his private data to Google, or did Google create an OS that automatically sent everyone's private data to their AI server for scanning without explicit consent?

izacus 2 hours ago | parent [-]

Google Photos does ask you for consent when you run it.

(It is, granted, a bit pushy about it and will ask multiple times when you run it with an intrusive dialog.)

GeekyBear an hour ago | parent [-]

Informed consent would require Google to inform you that their AI server will scan every photo you take with your device and report you to the police if it should detect (or hallucinate) something it doesn't like.

wildfireday2 3 hours ago | parent | prev [-]

[dead]

trvz 4 hours ago | parent | prev [-]

Google is not Apple. Apple customers expect the higher standard.

ribosometronome 2 hours ago | parent | prev | next [-]

The Pandora's box was already open and essentially no one noticed nor was there immense pushback that resulted in the features being removed. Photo scanning was already happening for both Android and Apple for the purpose of image search.

trollbridge 6 hours ago | parent | prev | next [-]

I thought the client side scanning was to protect children? If it suspects an image is bad, it blurs it and pops up a warning including a link to resources to go to for help.

Very different than trying to narc out users to the authorities.

Zak 5 hours ago | parent | next [-]

There were two different technologies. One was client-side scanning for known CSAM, which created a huge backlash and is described here: https://educatedguesswork.org/posts/apple-csam-intro/

The other is detection of images that may contain nudity, whether sent or received, when the owner/admin/parent enables the feature. It is relatively uncontroversial and is described here: https://support.apple.com/en-us/105069

EmbarrassedHelp 2 hours ago | parent [-]

The controversial part is having the system enabled by default with age verification required to turn it off, and having the system impact non-Apple/Google apps. The UK for example wants Apple and Google to forcibly enable nudity blocking on all devices in the UK, and they want the system to bypass app/DRM security to scan all content visible on a device.

pavon 6 hours ago | parent | prev | next [-]

That is what they actually deployed. They were planning on performing client-side scanning of all images uploaded to iCloud for CSAM and reporting it to the authorities, but backpedaled after public push-back.

michaelmrose 5 hours ago | parent | prev [-]

The original proposal was to use a visual hash designed to identify known bad CSAM images even if cropped or otherwise edited. Your computer would scan all your stuff for these images and report you to apple who would report you to the cops. This presented a number of issues.

Accidental false positives could lead to horrific outcomes up to and including oh look bob got shot by the cops for resisting.

It was possible to produce apparently matching innocuous images and then poison people's machines with them.Oops did you click on that picture of a tree have fun with the cops. Like an advanced form of swatting.

Although inspired by a desire to find CSAM Apple could be forced to scan for ANYTHING by repressive regimes including America and China.

Although initially targeting images client side scanning of messages is a pretty obvious next step. Again obvious good motivation exists and is completely justifiable who doesn't want to stop the next mass shooting or terrorist attack... and then we can basically use it to find people critical of the regime. Do remember we are presently prosecuting a political figure for a picture of sea shells and a guy in texas is rotting in prison for distributing political literature.

mmmlinux 6 hours ago | parent | prev | next [-]

Is it different than telling your therapist something in confidence and then finding police waiting for you in the lobby.

arcticbull 6 hours ago | parent [-]

Yes, in the sense that you have a legal doctor-patient privilege that binds what they can share with whom. There's not really an Apple cloud user privilege.

No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc.

AlexandrB 5 hours ago | parent | next [-]

A better analogy is a storage locker. AFAIK police need a warrant to search "your" storage locker even though it's on someone else's property. I don't see why data in the cloud should be any different. Pre-emptively scanning everyone's data is equivalent to officers rummaging through all the storage lockers in a facility "just in case" they find something illegal.

d1sxeyes 4 hours ago | parent [-]

It’s a bit more like requiring you to submit to a weapons pat down before you go to your locker I think.

busterarm 6 hours ago | parent | prev [-]

> No, in the sense that your therapist is still required to report you to the police in various situations where you pose an immediate threat to yourself or others, etc.

And therapists are legally mandated to report you if you told them you viewed or possessed CSAM.

Dylan16807 6 hours ago | parent | next [-]

No matter how or why? That seems like a terrible mandate.

mothballed 4 hours ago | parent | next [-]

They are mandated to report child abuse. It is the same story with doctors, if an abusive parent brings in a child for care they learn never to give the kid healthcare again after the doctor reports it. It is rooted in good intentions but the effect is it means abused children never get to see doctors, therapists, get a half-ass minimal homeschool instead of going to school, etc so that mandated reporters never enter the picture.

Dylan16807 4 hours ago | parent [-]

Reporting abuse the client was involved in has a compelling reason. That's different from hearing their client saw a picture of the abuse of a total stranger.

RajT88 2 hours ago | parent [-]

> That's different from hearing their client saw a picture of the abuse of a total stranger.

I get it, actually. It's totally possible the picture in question was not known to authorities prior. That's called due diligence to look into it.

Adults not looking into things or following up on things are how the system fails children if you read some accounts of people who were abused by their guardians. Horrifying stuff.

Dylan16807 43 minutes ago | parent [-]

It could theoretically help to flag those images. But that kind of submission should be anonymous. It shouldn't ruin the ability for someone to get therapy.

Mandatory reporting makes sense for situations you are connected to. And even then there's presumably good reasons not everyone is a mandatory reporter. This goes way beyond that, mandatory reporting once removed for someone that doesn't know a single person involved.

Hell, reporting someone for that doesn't even guarantee the images get looked into! If they didn't save history they're probably not feeling like going back to the site to demonstrate. Similar if it was sent against their will and they deleted it right away.

busterarm 5 hours ago | parent | prev [-]

> No matter how or why? That seems like a terrible mandate.

Honestly shocked that anyone would even say this, but even giving you the benefit of the doubt here -- the one case where I could imagine this might not happen would be if you're a police officer investigating such cases. But they also have their own therapists dedicated/trained in police-specific issues.

Dylan16807 5 hours ago | parent [-]

Even if someone went browsing for it, yes that's illegal but there's no benefit in their therapist reporting them for just visiting terrible websites.

But also there are definitely ways to get accidentally exposed. That's an absolutely awful thing to call the cops over.

busterarm 5 hours ago | parent [-]

I think you're demonstrating incredibly poor judgment here. CSAM is a crime with real victims. Even if your patient came across it innocently, someone is out there intentionally distributing it and that needs to be investigated.

AnthonyMouse 4 hours ago | parent [-]

Have you considered the implications of what you're saying?

A whistleblower goes to a therapist, stressed out over their pending decision to reveal official misconduct. They've been investigating ways to post something on the internet that can't be immediately taken down by the corrupt government officials they want to expose. They express their discomfort, in confidence, to their therapist, about using something they've discovered is also used for CSAM.

You think it's a good thing for the therapist to be required to report this? Should they report that the patient admitted to viewing CSAM with no context so the whistleblower gets investigated and arrested, or should they provide the context -- that the patient is about to expose the corruption of the government receiving the report?

For that matter, consider what it does when someone is actually a pedophile. They find out that if they try to seek therapy to address their perverse attraction to kids, the therapist isn't allowed to keep their confidence and they'll be arrested, so instead of seeking professional help, they keep abusing kids. Is that the result we wanted? There is a reason doctor-patient confidentiality was a thing.

mothballed 4 hours ago | parent | next [-]

I used to have a good friend that was a stripper (I promise, I wasn't the client...). Some of her biggest customers were people that wanted therapy without the paper trail of going to a licensed therapist. This is a big thing for pilots as well, since their health records and mental care are intensely scrutinized. A stripper will provide comfort, verbal relief, and physical love for $100 hour you can tell them anything and their reputation is bad enough no one will bother to believe them if they say something bad about you.

gausswho 4 hours ago | parent [-]

Makes you wonder if there's a strip-joint where they're all licensed therapists, but non-practicing.

busterarm 4 hours ago | parent | prev [-]

Well, in the jurisdictions that I care about the courts and lawmakers have already decided this and the legal requirement is to report.

If its your license to practice on the line you know what choice you're going to make.

rootusrootus 2 hours ago | parent [-]

Therapy will become progressively more useless as people avoid it because therapists are required to report anything they hear which might be a crime.

freehorse 4 hours ago | parent | prev [-]

That's not generally true. From [0]

> Across most states, viewing CSEM alone is generally not a mandated-reporting trigger; reporting becomes obligatory when disclosures involve an identifiable child being abused or used to produce material.

> California’s CANRA imposes a distinct duty to report electronic access (download/stream) with identifying patient information, upheld against privacy challenges based on compelling state interest.

[0] https://www.psychiatrictimes.com/view/mandatory-reporting-ch...

winningChild 5 hours ago | parent | prev | next [-]

[dead]

FireBeyond 5 hours ago | parent | prev [-]

I still also totally don't get their policy.

Trying to avoid false positives by not firing until a threshold was hit (was it 20 images?) seemed insane from a PR position... rightly or wrongly, all it would take would be the wrong court case and you can see the headlines:

"Apple says users can have up to 20 CSAM images on their phone before they'll tell police"

xphos 5 hours ago | parent | next [-]

Imagine you have pictures of someones baptism and the kid was nude. Is it CSAM? I think the program would have to say use but morally I'd say no. The issue with client scanning is it has to assume the worst, or they are than liable. If its the person has 20+ different baptism of nude babys well huh that actually might be CSAM because the context of how that concentrated photos implies but even than its hard what if that person actually has 20 God Children its less crazy than one thinks... Especially if they have multiply phones from a single baptism.

You might not like pictures that way but honestly I think more important in procescuting CSAM is to go after the large sources of CSAM generation. Its trafficing in East Asia, and in Europe. I think weirdly America actually produces less CSAM in general because Americans are lot more off put by Sex than most other cultures. Abuse definitely happens in the US but making policy decisions like this produces bad policy.

Does iCloud rehost the photos to other people I don't really know because I use andriod tbh. If they are being rehosted (I assume to members of your contacts) that can be problematic but I think honestly the issue a lot more complex than just protect the children which the source of critic is a lot attacks against apples are coming from

nylonstrung 5 hours ago | parent | next [-]

> America actually produces less CSAM in general because Americans are lot more off put by Sex than most other cultures

The US has the largest pornography industry in the world by a massive margin, and the largest consumption of online pornography per capita

Meanwhile should we be surprised that CSAM production is higher in countries like the Philippines that have very weak digital policing, abject poverty, high numbers of street children etc?

dgellow 5 hours ago | parent | next [-]

The US is pretty extreme, you have at the same time easy access to all the pornography you want (unless you’re in states that require age verification), and also a very prude culture. It’s not the only place with such contradictions though

RajT88 an hour ago | parent [-]

Texas is particularly wild to me as a midwesterner. All the bibles and megachurches and all that and... They have strip clubs freaking everywhere! Really weird zoning laws as well - I recall seeing a strip club across the street from an Office Depot, next to a deli. The suburbs where I live, you have to go to a seedy part of town so you feel good and ashamed about it. lol

Some TV shows rightfully take aim at this contradiction (recently: The Hunting Wives).

mothballed 4 hours ago | parent | prev [-]

The age of consent in the PI was like 14 until a couple years ago when they changed it to roughly match international norms, in their culture it was considered consensual rather than abuse until very recently.

FireBeyond an hour ago | parent [-]

You can get married with parental or a judge consent at any age in Mississippi, New Mexico, and Oklahoma.

You can get married at 15 in Hawaii and Kansas.

You are "grandfathered" into sexual consent (not as in "marital rape" but "no longer statutory rape") when you do so.

So we may not want to be lording it too much on "international norms", particularly given that most of the marriages that happen at those ages are not "young couple got pregnant" but "older man in conservative/religious community".

d1sxeyes 4 hours ago | parent | prev | next [-]

The proposed mechanism was hash matching against known CSAM images, so the baptism photos would not trip the filter because they wouldn’t be hash matches.

wildfireday2 3 hours ago | parent [-]

But AI algorithms actually deployed by other cloud services do not. You’d hope that they wouldn’t flag a baptism or bris photo but currently fielded systems are flagging doctor-patient medical photos and have ruined lives, so.

dgellow 5 hours ago | parent | prev [-]

> Imagine you have pictures of someones baptism and the kid was nude. Is it CSAM?

In some countries it is as far as I’m aware

fortran77 5 hours ago | parent | prev [-]

The practical problem is, without a threshold, they'd have an order of magnitude more false positives than 'real' detections, making the system useless.