PII in AI: what it means and how to stop it leaking

PII in AI means any personal detail (a name, an address, a health note) that gets fed into a model, plus anything the model can guess from context it was never directly told. The single biggest risk isn't a hacker. It's raw personal data quietly slipping into a prompt, a training set, or a chatbot's memory. The one thing to do right now: stop raw PII reaching any model, full stop.

Personal data enters AI systems through a few main doors:

• Training and fine-tuning data
• Prompts typed by staff or customers
• Embeddings and vector databases (used for search and memory)
• RAG systems (Retrieval Augmented Generation, where a model pulls in your documents to answer questions)
• AI agents calling tools and other systems

Stanford HAI frames this well: privacy risk in AI is a supply chain problem. It shows up at every stage, not just when data is stored. Treat it that way and you'll catch far more than a firewall ever will.

What counts as PII in AI: direct, indirect and inferred identifiers

PII splits into three buckets. Get this wrong and you'll miss the real risk.

Direct identifiers are the obvious ones. Name, email, phone number, national insurance number, passport number. Anyone can spot these.

Indirect identifiers are sneakier. A postcode plus a date of birth. A job title plus a company name. Alone, each is harmless. Combined, they can point to one person.

Inferred PII is where AI changes the game. A model can guess someone's income bracket from their purchase history. It can guess a health condition from search terms. Nobody typed that data in directly. The model worked it out.

That third bucket is the one most privacy policies never contemplated. Nobody consented to an inference.

Think of it like a jigsaw. One piece means nothing. Enough pieces and the picture is obvious.

Common enterprise examples:

• CRM fields: customer name, deal notes, phone numbers
• Support tickets: "my daughter's account", medical details, addresses
• Attachments: PDFs, screenshots, spreadsheets pasted into a prompt
• Call transcripts: voices, names, account numbers spoken aloud

Every one of these can end up inside an AI workflow without anyone meaning it to.

How PII gets into AI systems: training, prompts, RAG and agents

PII doesn't need a door left open. It finds five different ones.

1. Training and fine-tuning. Models trained or fine-tuned on scraped web data, support logs, or old email archives can absorb personal data without anyone checking first. Once it's baked into the model's weights, removing it is nearly impossible.

2. Prompt-time leakage. Staff paste customer emails into a public chatbot to "tidy up" a reply. That is shadow AI, meaning tools used outside the approved system, and it is one of the fastest-growing risks in any business. The pattern is always the same: no policy, no approved tool, so people reach for whatever is open in a tab. We wrote up what an unmanaged AI account actually costs after one shared login ran up $5k in an hour.

3. Embeddings and vector stores. When you turn documents into embeddings (numerical representations a model can search), the original text can often be reconstructed from those numbers. A leaky vector store is just as risky as a leaky database.

4. RAG retrieval. RAG systems pull relevant documents into a prompt before the model answers. If your retrieval system doesn't check sensitivity first, it may expose a customer's medical history to unrelated queries.

5. Agent and tool calls. AI agents that book meetings, update records or send emails pass PII between systems in ways nobody mapped out. The output path is the blind spot. Everyone checks the input. Almost nobody checks where the answer goes next.

That fifth door is the one that opens as you scale. A single assistant is easy to watch. A network of agents working across your operations is not, unless the permissions were designed in before the agents were built.

IBM's analysis puts it plainly: AI breaks the old privacy playbook. Data no longer sits still in a database waiting for a quarterly scan. It moves, it gets rewritten, it gets recombined. Controls built for data at rest cannot keep pace with data in flight.

Detecting and classifying PII in AI contexts

You cannot protect what you cannot see. Detection has to reach everywhere the data lands: spreadsheets, PDFs, chat logs, and the embeddings sitting in your vector store.

Here is the problem. Regex (pattern matching, like "find anything shaped like an email address") catches the obvious formats and misses the rest. It will not spot a name in a sentence. It will not spot a health condition described in plain English. Static DLP (data loss prevention) tools were built for files sitting on a server, not for text moving through an API call in milliseconds.

So detection has to work on meaning, not shape. That means models trained to recognise PII by what it is, not by what it looks like.

Azure's Language service is a good example. It detects and redacts PII across text, chat transcripts and documents before any of it reaches a model. The pattern that matters is where it sits: in front of the model call, scanning every request on the way through.

Good detection needs to happen in three places:

• Structured data: CRM tables, spreadsheets, databases
• Unstructured data: documents, emails, support tickets
• Embeddings: sensitivity tags attached to each chunk before it's turned into a vector

Tag sensitivity at the chunking stage, not after. Once text is embedded, going back to add metadata is far harder than attaching it up front. This is a knowledge architecture decision, not a security afterthought, and it is much cheaper to make once.

Protecting PII in AI pipelines: anonymisation, dynamic masking and policy gates

Detection tells you where the risk sits. Protection is what stops the leak. Four steps, in order:

1. Classify. Tag every data source by sensitivity: public, internal, confidential, restricted.
2. Set policy. Decide what each classification is allowed to do. Restricted data never leaves your systems unmasked.
3. Enforce at delivery. This is the moment data actually reaches the model. It's the last checkpoint, and the most important one.
4. Trace. Log every decision so you can prove what happened, when, and why.

Think of the kitchen pass in a restaurant. Nothing leaves the kitchen until the head chef has looked at the plate. Delivery-time enforcement is your kitchen pass for AI. Most teams put their controls in the storeroom instead, then wonder how the wrong plate got out.

The pattern to copy is the anonymise, LLM, de-anonymise sandwich. Presidio, the open-source PII framework that started at Microsoft, is the reference build. It swaps real names and numbers for placeholders before the prompt goes out, then puts the real values back into the answer that comes home. The model never sees the name. It sees "Person A" or "Customer 12".

The clever bit is that placeholders are consistent. The model still knows Person A said X and did Y, so the answer stays coherent. You get the quality without ever handing over a real identifier.

Dynamic masking pushes the same idea back to retrieval time. There is useful background on AI output preprocessing techniques for data scientists if you want the mechanics. The check is simple. Before a RAG system builds the context for an answer, it asks whether this user, or this agent, is allowed to see this chunk. If not, mask it or drop it before the prompt is ever assembled.

For that check to run at speed, the sensitivity tag has to already be on the chunk. Attach it when you index, not when you retrieve. Otherwise a human ends up in a loop that should never have needed one.

Per-agent access checks matter just as much once more than one agent is running. The agent that handles billing has no business pulling medical notes, even when both sit in the same underlying system. Give each agent the narrowest access its job needs.

A few operational basics round this out. Devwiz has a plain-English walkthrough of AI data security basics for businesses if any of this is new to your team.

• Retention. Don't keep raw prompts and responses longer than you need to. Set an expiry.
• Storage. Encrypt anything containing personal data, at rest and in transit.
• Output handling. Check what the model sends out, not just what goes in. Outputs can leak PII too, especially in agent chains.

Pro Tip: Run a de-anonymisation test monthly. Send a masked prompt through your pipeline and check the placeholder correctly maps back to the real value. If it doesn't, your sandwich has a hole in it.

What should an audit see? Proof of lineage. Trace one piece of data from its source, through every transformation, into an embedding, through a retrieval event, past a policy decision, to the agent that used it, out to the final answer. Miss one link and you cannot prove the controls worked. Saying they worked is not proof.

Governance, compliance and risk assessment for PII in AI

Regulators are catching up fast. They expect documentation, not good intentions.

NIST's guide to protecting confidentiality of PII asks organisations to sort data by risk level and record the controls applied to each level. Saying "we protect PII" is not an answer. You have to show which control sits on which data, and why.

The EDPB's opinion on AI models makes a sharper point: a model trained on personal data may not count as anonymous, even if you can't easily extract a name from it. If the risk of re-identification exists, regulators expect a Data Protection Impact Assessment (DPIA), documented reasoning, and proof you considered the risk before deployment, not after.

The EU AI Act transparency rules that went live in August 2026 push in the same direction, and they reach any business serving EU users, not just EU companies.

US enforcement points the same way. The question being asked is whether a business had reasonable safeguards in place, not whether a breach was technically possible. If your AI system touches health data, financial data or anything about children, run a DPIA before launch. Do not wait to be asked why you did not.

Health data is the strictest case, and it shows what good looks like. Njin's write-up on running AI inside a healthcare practice sets out the requirements a clinical build has to meet: patient consent, data stored in the right jurisdiction, an audit trail for every AI-generated output, and a human reviewing each one before it enters the record. Most businesses will never face that bar. Building to it anyway is not wasted work.

What auditors actually want to see:

• Lineage: where did this data come from, and where did it go?
• Purpose: why was this data processed this way?
• Retention: how long is it kept, and why that long?
• Access logs: who, or which agent, touched it, and when?

Build this in from day one. Retrofitting lineage tracking after a regulator asks for it is miserable work, and it never gets cheaper.

A PII in AI runbook you can start this week

You do not need a six-month project to start. Here is a runbook you can begin this week.

1. Discovery run. Scan your CRM, support tool, documents and vector stores for direct identifiers. Quick wins first: emails, phone numbers, national insurance numbers.
2. Block raw prompts. Stop staff pasting unmasked customer data into any AI tool. Add anonymisation middleware in front of your main model.
3. Enforce at retrieval. Add a sensitivity check before your RAG system assembles context for any answer.
4. Run audit tests. Try prompt injection attempts, test whether RAG retrieval leaks restricted chunks, and verify de-anonymisation maps back correctly.

Start with your highest-traffic workflow, not your most sensitive one. Volume teaches you faster. Then take what you learnt and apply it everywhere else.

Where this belongs in the build

Here is the part most teams get backwards. PII controls are not a security project you run alongside the AI project. They are part of the AI system's design, and they belong in the same build.

The way we do it: your IP and your operating rules get encoded into an AI Operating System, a coordinated set of AI employees that each hold a defined role, a defined scope, and a defined set of permissions. We build those with Claude Code, which is what makes the controls practical rather than theoretical. Masking, retrieval checks and per-agent access rules are written into the agent definition itself, so an agent cannot reach data its role was never granted. The policy is not a document somebody has to remember. It is in the system.

That is also why the order matters. Explore, then Map, then Transform. You map where personal data actually moves through the business before you build anything on top of it. A generic automation tool bolted across your stack will happily move PII between systems, because nobody told it not to. An AI employee built against a mapped data flow will not, because the boundary is part of its job description.

Common mistakes, and what good looks like

Most teams still think about PII the old way: lock the database, tick the box. That falls over the moment a model can infer identity from fragments no database ever flagged.

The real skill is calibration. Redact too hard and the answers come back useless. Redact too lightly and you are exposed. You only find the line by testing, which is why the audit tests above are not optional.

Aim for three milestones in order: stop raw PII reaching models, add policy gates at delivery, then automate your audits so nobody has to check by hand.

Build governed AI systems without the guesswork

There are other routes. Hire a data engineer to bolt detection tools on. Ask your IT team to work out masking on the side. Both get there eventually, if you have months spare and nobody minds learning on the job.

We do it differently. We map your real data flows first, then build the controls into the AI employees themselves, so governance ships with the system instead of chasing it. Our assessment shows where personal data moves through your workflows and what needs fixing before you scale.

If you run a program or consultancy earning $1M or more and you want AI agents handling client work without leaking client data, the 90-day program builds that whole system with you, hands-on.

Start with the assessment. It takes a few minutes and it tells you where your data is going right now.

Sources

• Data privacy and foundation models, Stanford HAI
• AI privacy analysis, IBM
• PII detection in Azure Language, Microsoft Docs
• EDPB opinion on data protection aspects related to AI models, EDPB
• Guide to protecting the confidentiality of PII, NIST
• Presidio, open-source PII detection and anonymisation