Privacy
Handling information that lets someone be identified
- Privacy is about information that can identify a specific person. That covers not just names and numbers, but also fragments that become identifying once combined.
- With AI, the trouble spots are three: when data goes into training, when it's typed in during use, and when it slips into an answer.
- Removing a name is not enough on its own. Piecing together what's left often makes someone identifiable again.
- Once information is baked into training, it's hard to pick back out. That's the biggest difference from a paper directory.
- The basic response is to collect less, tie usage to a narrow purpose, and process on-device wherever possible.
Contents
1The analogy
You put together a membership directory for a club. At first you meant to jot down just names and phone numbers. Then a bank account gets added for handling dues, a clothing size for ordering team gear, a birthdate for sending birthday wishes. Once it's all on one page, the directory has become a different object than it started as.
That raises questions. Who gets to see this directory? Does someone checking the member list also need the bank accounts? Would stripping the names be enough? If sizes, birthdates, and neighborhoods stay together, anyone who knows the group can still figure out who's who.
Someone quits the club, so their row gets deleted — except nobody knows where last year's photocopied directory ended up. These are the same questions that come up around AI and privacy.
2In detail
Fragments change character once combined
Each piece looks harmless on its own. Neighborhood, weekly schedule, favorite sport, device type — none of these alone says who you are. But once three or four of these overlap, the pool of candidates shrinks fast. Add a few more, and it narrows to one person.
That's why current rules don't just ask "is this one field personal data" item by item — they ask "does this combination let someone be identified." The same field can change character depending on what else it's sitting next to.
AI is very good at exactly this kind of combining. Finding patterns in scattered fragments is what it does naturally, so it can narrow down an individual even from data that looks harmless to a person.
Three places it goes wrong
The first is when data goes into training. Scrape up public posts and photos for training material, and people's stories come along with them, often for a use the poster never intended — which is exactly the point of dispute.
The second is while typing during use. Paste in a whole document for better help, and information about people who have nothing to do with the request comes along for the ride. How much of that sticks around varies by service.
The third is when it leaks into an answer. Content that showed up often enough in training can resurface in an answer, and information typed into one conversation can end up flowing somewhere another user later sees it.
Removing a name doesn't make it disappear
The most common response is stripping out names and numbers. This is called pseudonymization — it means the key needed to reverse it still exists somewhere. That's different from anonymization, where reversal is meant to be impossible.
The trouble is re-identification without the key. If what's left is detailed enough, matching it against other data can narrow down who it is. The more detailed the data, the more useful it is — and the more it can be re-identified. You can't max out both at once.
So in practice, the real work is deciding together how much to blur, which fields to drop entirely, and who gets to see the rest under what conditions. It's not a matter of picking one processing method — it's a matter of designing a whole procedure.
Once it's in, it's hard to pick back out
Delete a row from a directory, and that person's information is gone. A trained model doesn't work that way. The data isn't stored as a file — it's spread thin across an enormous number of values, so pulling out one person's trace isn't simple.
So current methods amount to workarounds: dropping the problem data and retraining, filtering an answer before it goes out, or blocking that data from getting in at the front end in the first place. Methods for making a model forget one specific piece of information are being studied, but they're not in wide use yet.
Copies are a lingering problem too. A model gets copied and run in many places, and an answer already sent can't be pulled back — the same problem as not being able to collect every photocopy of a directory already handed around. Last checked: 2026-09.
Collect less, tie usage to a narrow purpose
The most reliable defense is collecting less from the start. Take only what the task at hand actually needs, and delete it once that need is gone. Information never collected can't leak.
Next is tying usage to a narrow purpose — never using a bank account collected for dues to send event invitations, the same idea. A record of who viewed what and when makes it possible to check whether that promise holds.
Third is choosing where processing happens. Processing on-device and sending out only the result means the original never leaves. Combining what each device learned, instead of pooling everyone's raw data, comes from the same idea — though what piles up on the device, and who can reach it, still needs its own look.
3More precisely
Rules across different countries tend to share a common backbone: collect data for a defined purpose, collect only as much as that purpose needs, don't use it beyond that purpose, delete it once it's no longer needed, and disclose what's collected and how it's used. The details and exceptions vary by country, and how AI training fits inside this backbone is still being worked out.
There's a tension worth naming too. Tying data usage too narrowly can make it harder to check for bias — spotting which group is getting worse outcomes requires being able to identify that group in the first place. Collecting less and checking for fairness pull in different directions, and neither one is a complete answer by itself.
The analogy breaks down in places. A directory can be edited row by row, but a model can't easily have one person's information picked out and removed. A directory can log who opened it, while there's no easy way to count how far a model's answer has spread. And a directory is one physical object, while a model gets copied many times and runs in many different places at once.
Last verified: 2026-09
4Try it yourself
5Common misconceptions
It's easy to think removing the name means it's no longer personal data, but actually if what's left can be pieced together to narrow down a person, it's still treated as personal data.
It's easy to think asking to delete what a model learned actually deletes it, but actually pulling it back out is hard, so the usual fallback is retraining or filtering answers before they go out.
It's easy to think running on your own device is automatically safe, but actually the only real difference is that nothing leaves — what piles up on the device and who can access it still needs its own look.
7One-line summary
In shortPrivacy changes character the moment fragments combine, and once it's baked into a model it's hard to pick back out, so the choices made upfront — collecting less, tying usage to a narrow purpose — matter more than anything that comes after.
Spotted an error or have a better analogy? Suggest an edit · Last updated2026-09-02