Apple's NeuralHash Algorithm Has Been Reverse-Engineered (schneier.com) 86
An anonymous reader writes: Apple's NeuralHash algorithm (PDF) -- the one it's using for client-side scanning on the iPhone -- has been reverse-engineered.
Turns out it was already in iOS 14.3, and someone noticed:
Early tests show that it can tolerate image resizing and compression, but not cropping or rotations. We also have the first collision: two images that hash to the same value. The next step is to generate innocuous images that NeuralHash classifies as prohibited content.
This was a bad idea from the start, and Apple never seemed to consider the adversarial context of the system as a whole, and not just the cryptography.
Turns out it was already in iOS 14.3, and someone noticed:
Early tests show that it can tolerate image resizing and compression, but not cropping or rotations. We also have the first collision: two images that hash to the same value. The next step is to generate innocuous images that NeuralHash classifies as prohibited content.
This was a bad idea from the start, and Apple never seemed to consider the adversarial context of the system as a whole, and not just the cryptography.
What will we call it? (Score:5, Funny)
"Swatting" is what we call anonymously calling a SWAT team on people perpetrator doesn't like.
What will we call anonymously sending someone an image that has a "correct" hash collision with known child porn in the database?
"Appleing"?
Re:What will we call it? (Score:5, Funny)
Re: (Score:1)
Coring was my thought was well. I guess great minds think alike. Wish I had some mod points to share.
Re: What will we call it? (Score:1)
Re: (Score:2)
i hated not getting the joke above, and your comment just makes me feel worse. and i'm not even a z-gen to talk in this way, wtf is happening to me?
Re: (Score:1)
Re: (Score:2)
In China, you can Jinping someone by denouncing them to the state storm troopers.
"Appled" (Score:2)
We used to have "Slashdotted" for sites linked from stories going down in the old days, I say we use the same format here: Appled.
Re: (Score:2)
Or perhaps Cooked.
Re: (Score:2)
Goatseing
She got Goatsed
Re: (Score:2)
Cooking, after the glorious leader who seems to think this is a good idea.
Re: (Score:2)
What will we call anonymously sending someone an image that has a "correct" hash collision with known child porn in the database?
One image won't trigger it. There's a threshold before anything happens. Once that threshold is reached a human reviewer can see a low-res version of the flagged pics. If they are not actually the real thing there's no escalation.
I'm not defending the entire thing, but it's not so easy to entrap someone.
Who could have predicted this? (Score:2)
Re: (Score:2)
I wanted to have a pool on what date it would be cracked, but by the time it would have got going it would have been cracked:
https://slashdot.org/comments.... [slashdot.org]
I don't think that's how this works. (Score:5, Insightful)
This was a bad idea from the start, and Apple never seemed to consider the adversarial context of the system as a whole, and not just the cryptography.
Don't be silly, of COURSE they considered the adversarial context of the system. They just decided that either they didn't care that it was adversarial, or that their users were too lazy/dumb/unaware to do anything about it so they pushed it out anyway.
This is what happens when companies get this big. They stop analyzing their moves from a perspective of trying to give the customers what they want and/or ask for and start cooking up new ways to fuck with their customers. And sadly, in Apple's position today, no amount of fuckery will make their users turn away en-masse. Any that do are quickly forgotten in the profit stream coming from those that just accept it and keep buying Apple.
Re:I don't think that's how this works. (Score:5, Informative)
They just decided that either they didn't care that it was adversarial, or that their users were too lazy/dumb/unaware to do anything about it so they pushed it out anyway.
Actually, they sort of did: the clients don't get access to the hashes Apple is matching against. In fact, the clients can't even tell if they matched one of the hashes. Instead they get a "blinded" hashtable, where the row is determined based on the unencrypted hash, and the value is the encrypted "blinded" hash. A combination of the unencrypted hash and the encrypted hash is used to encrypt a "safety voucher" which is what allows Apple to view the original image.
Essentially, if it's a real match, Apple can decrypt the safety voucher because they know the original hash. If it isn't, they can't, and it'll produce garbage. The actual hashes Apple is matching against are kept secret, so knowing how to produce them doesn't really help the client know what Apple's working for.
Which, of course, is also a problem. There's no way to know what Apple's actually looking for. Are they looking for CSAM, or are they hunting down whoever leaked the last Apple prototype? We may never know.
Re: (Score:3)
Are they looking for CSAM,
CSAM? Given the context. I'm reluctant to google it.
Re: (Score:2)
Are they looking for CSAM,
CSAM? Given the context. I'm reluctant to google it.
Child Sexual Abuse Material [apple.com]
I had to search it to find it too, but did it with DDG and safe search...
Re: (Score:2)
It's apparently the new "official" term for what we might have previously just called "CP." I guess it's slightly more descriptive, so law enforcement and other people who track down those who traffic in it use it now. It's what Apple has referred to the images they're looking for since they announced this "feature" but it's not a term they invented. Like aitikin said, it stands for "Child Sexual Abuse Material."
Re: (Score:3, Interesting)
Also we need to remember that matching a hash doesn't automatically flag you as guilty of having CP, it simply flags the file for human review.
So if it's obviously not CP, it will get ignored. The real privacy implications come from building the capability to let them search for arbitrary things.
Re:I don't think that's how this works. (Score:5, Insightful)
So a human will review what may be none of their business. What if it's a private picture of an adult? Or maybe a scan of financial or medical papers?
Re: (Score:2)
Um yeah thems the beans
Re: (Score:2)
The real privacy implications come from building the capability to let them search for arbitrary things.
Evidently they already have the capability, and no one should have ever thought otherwise.
The privacy implications come from a private company to snoop through your device looking to report wrongthink to the government so you can be locked up for daring to take a photo of someone, the legality of which is often subjective. Is it an exploitative photo, or just a family snap of my daughter having fun in the nude? We'll let a unnamed low paid reviewer decide your fate!
I'm actually reminded of a case a few year
Re: (Score:2)
So if it's obviously not CP, it will get ignored.
And what if it's not so obvious? I'd like to think if I have nothing to hide I have nothing to fear, yet we live in a world where politicians in some countries have thought of classifying adult porn-stars with A-cups as "child pornography" (see Australian law from ~2008). We live in a world where a (I wish I could find references but my google fu is failing) a man in the USA had a professional porn actress come to his defense after he was accused of having child pornography despite the star being 19 at the
Re: (Score:2)
The images in the database aren't secret, they are all over the internet if you know where to look. 8chan trolls already have them, we know that because they regularly post them to 8chan to troll the mods.
Re: (Score:2)
Re: I don't think that's how this works. (Score:2)
Re: I don't think that's how this works. (Score:4, Insightful)
The question would be why are they reviewing user files in any form, automated or manual? This whole concept that it's on their hardware therefore they have full permission to do whatever they want with it is . . . more than a little disturbing to those of us that still think privacy is a valid concept that should still be given at least some place in the discussion.
Re: I don't think that's how this works. (Score:3)
Because it goes on their servers. This whole thing is icky to me. I do not like the solution Apple has come up with, but it is clear why they felt the need to come up with it. Having that type of material on their servers is undesirable in and of itself, not to mention that it would give law enforcement an excuse to cleave into their whole privacy pitch. Of course, the problem with privacy is that it is incompatible with the cloud and it allows bad actors to shield their misdeeds. For me, the benefits of pr
Re: (Score:2)
Privacy isn't incompatible with the cloud. Privacy is incompatible with the modern vision of tech companies. They've got the keys to the kingdom now that everybody and their grandma is storing shit in the cloud, and now they're going, "All that data you thought was yours? It's ours now. Thanks."
If we were living in a real world instead of a nightmare fueled by greed and corruption, "cloud" would mean somebody else handles the hardware side and the data you place out there is yours. They have zero god d
Re: I don't think that's how this works. (Score:2)
I agree with you, but the ownership problem is a legal, not a technical one. So the whole idea that you can run a server and not own any of the data on it only works if the legal system views things that way. Then there is the matter of different countries, states, etc.
As long as the state will hold cloud providers liable for the data stored on their machines, they are incompatible with privacy.
Re:I don't think that's how this works. (Score:5, Insightful)
Given the many other half baked ideas that Apple has proudly announced in the past, I wouldn't attribute to malice what can be explained by incompetence.
Or it's circles within circles (Score:2)
What if this was planned by Apple all along, and this is how they "prove" to the FBI, NSA, and everybody else that it's a bad idea? What if they left some hints for these hackers?
I don't think Steve Jobs had that kind of devious mind (I could be wrong), but Tim Cook ... now there's a guy who hasn't got Steve Jobs persona but wants to leave his mark, and maybe this is the kind of subterfuge he'd pull.
Re: (Score:2)
Re:I don't think that's how this works. (Score:4, Interesting)
A denial of service attack against the human review?
Re: (Score:1)
Also, since it's possible to modify an
Re: (Score:2, Insightful)
They're likely getting a lot of $$$ from overseas allowing governments to use this backdoor maliciously, while Apple can pretend otherwise. Given Apple's previous stance on security before this, it only makes sense if they're getting paid so much money to do this that it would counteract the people giving up on Apple for introducing backdoors.
Any evidence? Any information backed reasoning behind that? Anything verifiable that we, as a community, can digest and act upon? If so then please share it, otherwise there are words that describe people that peddle in foolishness.
Re: (Score:1)
because everything apple does is about its profits, and creating a backdoor that makes users think twice about buying apple products, will have been carefully evaluated and decided that whatever revenue stream this creepy backdoor opens will be worth more to the stock holders
Re: (Score:1)
It's worse than that: these companies are now so large that it's not just about profits; more and more it's about pushing some worldview. These companies are large enough that they can stand to have smaller profit and still push their pet agendas.
I mean seriously, these companies can spend billions a quarter on some pet project and it won't even matter to their bottom line; only Wall Street notices or cares when the companies only earn $49 billion in a quarter instead of $50 billion.
Re:It was never about "kids", it was about $$$ (Score:4, Insightful)
Because it's such a useless technology to have, when it can be so easily thwarted, so unreliable, and doesn't stop the creation of new CSAM.
What's the point? I'll tell you want the point is... for them to be able to look at any of your pictures at any time by simply sending a hash to your phone... remember, when the tool flags an image as possible CSAM, it gets sent for human review... that means that your personal photos that get flagged get looked at by another person who could leak them anywhere or do whatever they want with them, even though they aren't CSAM at all. This is super fucked.
Re: (Score:2)
I hope this gets ruled a 4th Amendment violation. I'd bet my left nut the government has been applying even more pressure in private than the already substantial public pressure. If Apple did this because they were threatened into it, that makes them acting as an extension of the government and
Re:It was never about "kids", it was about $$$ (Score:4, Insightful)
No, this is a stupid theory. Those governments already have access to user data, and trying to use this way to get into the phones is a ridiculous waste of time.
If you consider China, they force Apple to hold Chinese citizens' data within the country and available for law enforcement. They can go into anyone's data at any time, and perform scans on it right now if they want. Using the Neural Hash is a humongous distraction.
If a country could force Apple to add things into the database that's being scanned, they could just compel Apple to provide the data, particularly if it's in an iCloud backup, which Apple holds a backup key to. If iCloud Photos is turned off, the system doesn't run anyway.
Stop thinking this is nefarious from that point of view. You can disagree with the system all you like--there's plenty to be worried about--but this avenue of complaint is absurd. Any country with the power to compel Apple to do anything already has the means to scan data, or even better, just arrest someone out of the blue, plant the evidence, and pretend like their judiciary system works. They're not going to go through the extra fiddly steps of engineering some hashes so that the victim's phone rats them out for no reason.
Re: (Score:3)
Having access is not the same as being instantly notified when an image of interest pops up. Remember that all of the safeguards that apply to US images don't necessarily apply worldwide.
Re: (Score:2)
They could just force Apple to do server-side scanning of that data RIGHT NOW. The photo is only scanned if it's going to be synced with iCloud Photo. And because they don't care if it's secure or not, they can probably run a much faster, less restrictive algorithm to do the photo matching.
Like, this adds just a million extra steps for virtually no benefit at all. The only reason why this Neural Hash thing is interesting at all is that it purports to protect your privacy by only flagging certain photos that
Re: (Score:3)
You're quoting the marketing speak but the real reason this is done on-device is because it's computationally expensive and your CPU cycles are paying for it. They're collecting the iCloud photo backups and storying them unencrypted anyway. The only reason for the security theater around this is to have an excuse to do it with zero energy cost to themselves.
Re: (Score:1)
It was about virtue $$$ too. Don't forget that.
Too soon to make these claims, but... (Score:5, Insightful)
As suggestive as this report may be, each and every one of these tests would have to be performed on the actual released version of Apple's feature to have any validity for the claims being made. Based on Apple's announcements, this isn't a final released version. What this report does show is that there is already a committed adversarial network that is determined to find any such weaknesses and expose them.
Re: (Score:2)
Re: (Score:1)
Don't get me wrong, my torch and pitchfork are ready, but we've got 3 threads going ready to gather as angry apple villagers.
Anyone else notice that you don't see these people at the (I own an apple product) meetings, only at the outrage gatherings? Maybe they just work for the Torch and Pitchfork companies . . .
Innocuous images ... (Score:3)
The next step is to generate innocuous images that NeuralHash classifies as prohibited content.
Maybe one of Stephen Wright's:
I collect rare photographs... I have two...
One of Houdini locking his keys in his car... the other is a rare picture of Norman Rockwell beating up a child.
-- Stephen Wright
Re: (Score:2)
Trump tweeting the truth
Biden remembering where his keys are
No one getting offended after a joke is told about Obama
Bush getting an A after studying
Katy Perry looking cute
Fuck Apple (Score:2)
So just your reguar DCT? (Score:1)
> it can tolerate image resizing and compression, but not cropping or rotations
So basically the way every modern lossy image format compresses images.
Is it really just "compress the image down to literally just a hash"?
I wonder... it's likely that you could "uncompress" a blurry image from the "hash".
Unless it's piped through an actual hashing function afterwards. (Which, don't forget, can and do collide way more often than people think.)
Re: (Score:3)
. (Which, don't forget, can and do collide way more often than people think.)
Cryptographic hashes: No. Hashes for hash-tables: Yes and that is not a problem as long as it does not happen too often.
These two are pretty much fundamentally different. There are some semi-cryptographic hashes that are almost as good as a full crypto-hash for some uses, but are fast enough to be used in hash-tables. An example is SipHash. For the image-identification, we are definitely talking crypto-hash requirements.
Re: (Score:2)
it can tolerate image resizing and compression, but not cropping or rotations
So basically the way every modern lossy image format compresses images.
sorry? not at all?
Is it really just "compress the image down to literally just a hash"?
although you might argue (semantically) that a hash is some form of compression, they're fundamentally different things. compression aims for reconstruction, hashing aims for identification.
apple's thingy this isn't about compressing the image, but about comparing hashes of the image with a list of known hashes of known bad images. people just hacked it, proved that it can be abused and that their hash algorithm is crap (because they were able to produce not only collisions, but a pre-imag
Re: (Score:2)
But wouldn't this require you to know what to pre-image against? How would you get the target hashes in the first place?
Re: (Score:2)
you could make an educated guess and use any image that you would reasonably expect to be in the list.
depending on the feedback of the filter (i really haven't looked at the details), you could possibly test if any given image is in the list or not.
Re: (Score:2)
Is it really just "compress the image down to literally just a hash"?
Not exactly. From what they've said, they created an ML model that they trained on images and then slightly altered versions of those images and made it so that it recognizes as "the same" images that are basically the same. The first step is apparently a generic DCT but these are then fed into the ML, and the final step is taking the output from the ML and then hashing it.
I wonder... it's likely that you could "uncompress" a blurry image from the "hash".
If you had access to the final ML output, probably. But it's then put through an actual hashing process, and this presumably destroys en
Apple's response (Score:3)
Apple's response is, unsurprisingly, that it's not big deal and was expected.
https://www.theverge.com/2021/... [theverge.com]
They have created collisions? (Score:2)
That is probably the end of this stupid idea. Great research though!
So about that kiddie porn scanner... (Score:2)
Re: (Score:2)
So, that huge number they said was the odds of a hash collision was BS then, why am I not surprised. The Verge article says"but perceptual hashes like NeuralHash are known to be more collision-prone".
Still the main concern is the slippery slope of Apple searching people's phones and saying vaguely that the system will be 'extended'.
Re:So about that kiddie porn scanner... (Score:4, Informative)
Re: So about that kiddie porn scanner... (Score:2)
Kinda funny (Score:2)
I fondly remember 'unbreakable' copy protections from the last 40 years after having read the article.
I know, I shouldn't have read TFA but i had a weak moment.
Apple should just start telling the full truth (Score:2)
Apple seems to be doing that special type of Tesla like PR, which is to say no PR. The press gets statements, but never answers. Which wouldn't be a problem if their statements weren't self hurting half truths.
The next big scandal is predictible, everyone noticing they've been doing this for ages already.
https://appleinsider.com/artic... [appleinsider.com]
If they just bothered to tell people flagged files are checked by Apple itself and not directly passed to law enforcement it would blunt a lot of criticism. But actually res
I just wonder (Score:3)
Apple says they are aiming for a 1 in a trillion chance of a false positive, but suppose you are the unlucky winner of that particular lottery? What happens if you lose your job and become a pariah because of this false positive? Will Apple issue a public apology and admit that they were wrong/made a mistake? Will Apple create a job for you if you become unhirable everywhere else? Or will Apple just flip you the bird and leave you to live with the consequences of their mistake -- effectively retired and lucky to get a job flipping burgers until our robot overlords take over low skill menial labor jobs?
Re: (Score:2)
Re:I just wonder (Score:4, Insightful)
Yes, however, you're assuming that the poor people in the Philippines or other low wage country where they outsource all of this work to will actually have more than one or two seconds to look at the photo and then make a decision. We already have a pretty good idea of what it will look like based on current systems for content moderation. It'll just get dumped into the same queue as all other flagged content and SLAs will mean that they have, at most, a couple seconds to look at each image and make a decision.
My major problem with this in general, and I know I'm largely preaching to the choir here, is that it basically assumes guilt and you have to prove your innocence. Making matters worse is it's being conducted by a non-government entity, and there's no real oversight. Best you could probably do if you're the victim of a false-positive is sue Apple, who will be able to throw basically infinite resources at fighting it. Even assuming you do somehow win and Apple doesn't appeal, you're still "that kiddie porn guy" to everyone and it won't really matter that it was a false positive, that you can prove it, no one will take the time to listen. It's one thing if I work in an Apple store and you bring your computer to be fixed, and you have a bunch of kiddie porn files just sitting out on the desktop where it's basically impossible to miss, or I come across the files during the course of my work. The store calls the cops and they take over the investigation from there. For Apple to actively rifle through everyone's photos, on what is basically a giant fishing expedition without having to show probable cause or obtain judicial oversight in the form of a warrant... it is completely antithetical to the system of justice in the US where Apple is headquartered.
Re: (Score:2)
It's one thing if I work in an Apple store and you bring your computer to be fixed, and you have a bunch of kiddie porn files just sitting out on the desktop where it's basically impossible to miss, or I come across the files during the course of my work.
Been there, found some really suspect porn (not talking tiny kids, but definite jailbait). This was a different time, long before the era of See Something Say Something, so, not feeling particularly snitchy, silently replaced it with animals and goatse. Client never called to complain... Of course I also admin'd a caching Usenet proxy for a smallish rural ISP, and back then... wow.
Re: (Score:1)
This. Combined with the fact that the government already has a legal process to obtain a search warrant for your photos if they have a reasonable suspicion, this kind of proactive snitching behavior shouldn't be socially acceptable here. It's child porn today, something else tomorrow. It doesn't matter how many promises are made today, if someone else is in control of that system or the government tomorrow, they'll
Flag them anyway... (Score:2)
...I'm sure Apple will flag them all anyway & get a human being to look at the images to see if they really are kiddie porn.
That's gotta be a horrible job - looking at kiddie porn all day. How do people deal/cope with that?
Re: (Score:2)
Hav
Re: (Score:2)
Stop using iCloud? (Score:1)
Re: Stop using iCloud? (Score:2)
Re: (Score:1)
Still two big hurdles (Score:2)
Apple never intended the NeuralHash model to remain secret. Their security model only requires the blinded hash database remain secret.
There remains two big hurdles to this attack, the second one being insurmountable:
1. Need to break the encryption on the hash database to get the hashes to collide against
2. Apple uses a second, independently-developed algorithm to check for a hash collision before presenting the image for manual review; your attack image would have to collide with *both* the NeuralHash as w
Asians and petite people (Score:1)
Asian and petite people can get in trouble over their selfies whether they are in clothing or not. Because I bet some of their content will end up hashed as CP.
People who have a thing for Asian and petite people porn will get flagged by reviewers. Based on the stuff I've seen I can't tell their age. Don't think a lazy reviewer will bother with trying either.
No matter how you view it this will end badly. Humans are not good reviewers. Youtube is a good example of that. Since we're dealing with a very serious