Can an LLM Leak Its Training Data, and Why You Cannot Un-Train PII

There is a comfortable assumption behind a lot of LLM projects: that a model trained on a pile of text learns patterns from that text, not the text itself, so whatever sensitive data went in is safely blurred into statistics on the way out. It is a reasonable intuition. It is also, for the specific question of whether a model can reproduce data it was trained on, wrong often enough to matter.

This is the first of a three-part series on why PII redaction belongs around LLMs, and it is the part that has to stand on evidence rather than worry. The claim is narrow and it is testable: large language models memorize some of their training data verbatim, that memorized data has been pulled back out by researchers, and once it is in the weights there is no reliable way to take it back. If that is true, then the only dependable control is to keep the sensitive data out of the training set to begin with. The rest of this post is about whether it is true, and where the honest limits of the claim are.

Yes, and it has been demonstrated

The cleanest evidence is not a theoretical attack, it is researchers recovering real strings from real models.

In 2021, Carlini et al. showed in Extracting Training Data from Large Language Models (USENIX Security) that GPT-2 could be prompted into emitting verbatim sequences from its training data, including a named individual’s full contact details: name, email address, phone number, and physical address, all recovered from the model rather than from any database. The data was not paraphrased or approximated. It was the original text, reproduced.

That was a smaller, openly studied model. The natural objection was that frontier production systems, trained on far more data with more aggressive deduplication, would wash this out. In late 2023, Nasr, Carlini, et al. answered that directly in Scalable Extraction of Training Data from (Production) Language Models . Using a simple divergence attack (asking a model to repeat a word indefinitely), they caused a deployed, aligned production chatbot to break form and emit long runs of memorized training data, again including personally identifiable information. The takeaway was not that one provider was careless. It was that memorization survives scale, deduplication, and alignment, and that a cheap query can surface it.

Between those two results, Carlini et al. also quantified when memorization gets worse, in Quantifying Memorization Across Neural Language Models . Three factors increase how much a model memorizes: larger models memorize more, data that is duplicated in the training set is far more likely to be regurgitated, and longer context makes extraction easier. The middle factor is the one most relevant to a team building its own dataset. A customer record, a clinical note template, or a signature block that recurs across thousands of documents is exactly the kind of repeated content models latch onto.

What this does not say (the honest part)

A proof-driven argument has to be just as clear about where it stops, because the failure mode of this topic is overclaiming, and overclaiming is what gets a security argument dismissed.

Extraction can produce fabricated PII, not only real PII. A model asked for “a patient named John” will happily generate a plausible name, address, and record number that belong to no one. That output looks identical to a memorized real record, and you usually cannot tell which is which from the text alone. So a single extracted string is not, by itself, proof that a specific person’s data was in the training set. The demonstrated risk is statistical: across enough extraction, some fraction of what comes out is genuinely memorized real data, verifiable against known sources. It is real, and it is also not “every name the model says is a leak.”

Proving whether a specific person was in the training set is hard at scale. The tool for that question is membership inference: given a record and a model, decide whether the record was trained on. It works reasonably on small models and overfit ones. On frontier-scale LLMs it weakens considerably. Recent work such as Duan et al., Do Membership Inference Attacks Work on Large Language Models? found that many membership inference attacks perform close to random guessing on large models, because models that see enormous, near-deduplicated corpora for very few epochs do not overfit individual records the way smaller models do.

It is tempting to read that as reassuring. It is the opposite. If you cannot reliably tell whether a given person’s data was memorized, you also cannot reliably audit a trained model and certify that it was not, and you cannot answer a data-subject deletion request by checking. The weakness of membership inference does not lower the risk, it removes your ability to measure it after the fact. That is the bridge to the real problem.

Why you cannot un-train it

Suppose memorized PII is in your model. What are your options for removing it?

Retrain from scratch without the data. This works, and for a large model it is also the most expensive option there is. For a frontier model it can mean weeks of compute and a budget that makes “just retrain it” a non-answer for most teams, every time a new record needs to come out.

Machine unlearning. The active research field of machine unlearning aims to surgically remove the influence of specific training examples without a full retrain. It is a genuinely promising area, and it does not yet offer what a compliance program needs: a verifiable guarantee that a particular person’s data is gone from the model, with no residual that a future attack could recover. Approximate unlearning methods can reduce a model’s tendency to reproduce a target, but demonstrating that the information is truly and permanently removed (rather than merely suppressed for the prompts that were tested) remains unsolved. Relying on it today means trusting a removal you cannot confirm.

Output filtering. You can put a redaction or moderation layer on the model’s responses, which is worth doing as defense in depth. But it is detection applied after generation, and detection is probabilistic: it catches a lot, not everything, and it does nothing about the data still sitting in the weights waiting for a phrasing the filter did not anticipate. (This is the same reason provider guardrails are a second layer and not the boundary for prompts.)

Notice the asymmetry. A miss in a real-time redaction pipeline is a one-time exposure: you find it, tighten the policy, and move on. A miss in training data is encoded into the weights and stays there for the life of the model, retrievable by anyone with access. There is no patch for a weight.

The keystone: redact before training is the only dependable control

Put the two halves together. Detection at inference is probabilistic, so an output filter cannot promise to stop leaks. Removal after training is unreliable, so you cannot count on pulling a memorized record back out, and you often cannot even confirm whether it is in there. Every control that operates after the data has been trained on inherits one of those two weaknesses.

The one control that does not is the one that runs before. If a sensitive value never enters the training set, there is nothing to memorize, nothing to extract, nothing to unlearn, and nothing to certify the absence of later. You are not trusting a filter to catch it on the way out or an unlearning routine to scrub it after the fact. You removed the dependency entirely by handling it at the only point in the lifecycle where removal is clean.

That is not a claim that upstream redaction is perfect. Detection is probabilistic wherever it runs, and you are responsible for validating its output against your own data. The point is narrower and it holds: upstream is the only stage where a value you successfully remove is actually gone from the model, instead of merely harder to surface. Every downstream control is mitigation on top of a problem you have already baked in.

What that looks like in practice

This is the part Philterd is built for. Phileas is the open source redaction engine, and Philter is the self-hosted API around it; both run inside your own environment, so the raw training data never leaves your boundary on its way to being cleaned. For training-data work specifically, two things change relative to a typical real-time policy:

• Bias toward over-redaction. When the consumer is a model that will memorize what it sees, the costs are not symmetric. A redaction that is slightly too broad costs you a little training signal. One that is too narrow costs you an identifier baked into the weights permanently. The training-data policy should be tuned accordingly.
• Pseudonymize where relationships matter. If the value of the corpus is in its structure (that two records describe the same patient, that a thread is one conversation), replacing every identifier with a generic token destroys what the model was meant to learn. Consistent pseudonymization swaps each real identifier for a realistic stand-in and maps the same source value to the same replacement everywhere, so the relationships survive while the real identities do not.

The detection itself leans on PhEye ’s NLP models for the entities a pattern cannot find (names, organizations, locations, clinical terms), which is most of what hides in unstructured text. And there is a reason we are comfortable making this argument: those models were trained on synthetic and public data, never on customer records , so the detectors finding PII in your data were not themselves built by absorbing someone else’s.

If you are fine-tuning or training on proprietary or regulated data, the AI training data use case lays out the approach, and we run it with teams as a focused AI training-data de-identification engagement inside your own cloud. For the retrieval-pipeline version of the same problem, see building a privacy-aware RAG system .

The takeaway

The honest answer to “can an LLM leak its training data” is yes, demonstrably, with the caveat that not every extracted string is a real person and that proving any single person’s presence is hard at scale. The honest answer to “can I take the PII back out afterward” is: not reliably, not verifiably, and not cheaply. Those two facts point the same direction. The dependable place to handle PII in training data is before the data is ever trained on, because it is the only place where removing it actually removes it.

The next posts in this series take it from there: redacting both sides of the LLM (inputs and outputs) in production, and the clean-room process for turning raw, identifier-laden data into a training set you can measure before it leaves the room.

If you are about to train or fine-tune on data that contains PII or PHI, that is exactly the right moment to talk: take a look at the AI training data use case or get in touch.