Philterd

Contact Us

Tell us about your stack and the privacy problems you're trying to solve. We typically respond within one business day.

Prefer email? support@philterd.ai

Please do not enter PII or PHI in this form. If you need to share an example, use a sanitized one.

← All posts

Why Your LLM Provider's Built-In Guardrails Are Not Enough

By Philter · Redaction

Every major LLM provider has a privacy story. OpenAI has content filtering and a moderation endpoint. Anthropic has usage policies and safety classifiers. Google offers DLP integration. AWS points you at Comprehend for PII detection. If you are a technical evaluator or a compliance officer trying to adopt an LLM responsibly, it is tempting to read that list and conclude the providers have the PII problem covered.

They do not, and the reason is structural rather than a matter of how good any one filter is. Every one of those guardrails runs in the wrong place: on the provider’s infrastructure, after your data has already left your network. By the time a provider’s filter has an opinion about the Social Security number in your prompt, the prompt has already crossed the wire, been terminated at the provider’s load balancer, and been handled by a third-party system you do not control. The exposure already happened. The filter is commenting on it after the fact.

Filtering on the way out is not the same as never sending it

There is a category difference between two sentences that sound similar:

  1. “The provider filters PII out of responses and flags it in prompts.”
  2. “The PII never reaches the provider.”

The first is a feature that operates on data the provider already has. The second is a property of where the data flows. Only the second one actually keeps the sensitive data inside your perimeter.

Consider what happens to a prompt in the first case. Your application assembles a prompt that includes a patient name, a date of birth, and a record number. It opens a TLS connection to the provider and sends the prompt. The bytes are now in the provider’s environment. They may be buffered, logged for abuse detection, retained for some window, and processed by whatever guardrail the provider runs. Even if that guardrail correctly identifies the PII and refuses to act on it, the data was still transmitted to and received by a third party. For a regulator, “we sent it but they promised to filter it” is not the same posture as “we never sent it.”

Where the redaction happens determines your compliance posture

This is not a philosophical nicety. The location of redaction is the thing that sets your compliance boundary, and the boundary is what auditors examine.

  • HIPAA. A hosted LLM provider that receives protected health information is handling PHI on your behalf, which puts you in business-associate territory and means you need that relationship papered and the data accounted for. If the PHI never reaches the provider, the provider is not in your BAA chain at all. (We wrote about why architecture, not paperwork, is what removes a vendor from that chain in Why Philterd doesn’t sign a BAA .)
  • PCI DSS. Cardholder data that crosses into a third-party system pulls that system, and the path to it, into your assessment scope. Redacting before the data leaves keeps the provider out of scope. Redacting after it arrives does not.
  • Government and FedRAMP/ATO. The authorization boundary is drawn around the systems that handle the data. A system that receives sensitive data is inside the boundary whether or not it later discards it.

In each case the question an auditor asks is not “does the provider have a filter,” it is “does the sensitive data reach the provider at all.” Provider-side guardrails cannot change the answer to that question, because they run after the answer is already yes.

“But they promise they don’t train on my data”

The most common objection is that the provider contractually commits not to train on your data, or to delete it after a retention window. Those commitments are real and they matter. They are also the wrong kind of guarantee for this problem.

A contractual promise is enforced after a breach, by lawyers, on the assumption that you can detect the violation and prove it. An architectural guarantee is enforced continuously, by physics, because the data was never transmitted. When the sensitive value never leaves your network, there is nothing for the provider to train on, log, leak, retain past the agreed window, or expose in an incident, regardless of what any contract says or how well the provider honors it. You are not trusting a policy. You have removed the dependency on the policy.

This is the same reasoning behind why API-based redaction is a security antipattern : the moment your protection depends on sending sensitive data to someone else first, you have already lost the property you were trying to keep.

The fix: redact inside your perimeter, before the prompt leaves

The architecture that actually holds is to redact prompts inside your own network, before they are sent anywhere. The provider receives an already-clean prompt and never has access to the original sensitive values.

The Philter AI Proxy is the drop-in way to do this. It speaks the OpenAI, Anthropic, and Amazon Bedrock wire protocols, so your existing SDK code does not change. You point the client’s base URL at the proxy instead of at the provider, the proxy strips PII and PHI from the prompt according to your policy, and then it forwards the cleaned request upstream:

# Before
client = OpenAI(api_key=KEY)

# After: point at the proxy. Same client, same code.
client = OpenAI(
    api_key=KEY,
    base_url="https://philter-proxy.internal/v1",
)

The proxy runs inside your environment. The redaction happens before the request crosses your network boundary, so the provider sees a prompt with the names, account numbers, and record identifiers already replaced. The provider’s own guardrails are then a redundant second layer on data that was never sensitive to begin with, which is exactly where you want them: as defense in depth, not as your only line.

This is the same pattern we use in Building a HIPAA-compliant medical chatbot , and it is the foundation of the AI and LLM use case more broadly.

The takeaway

Provider guardrails are worth having. They are not worth relying on, because they cannot answer the only question that sets your compliance boundary: did the sensitive data reach the provider. The only way to make that answer “no” is to redact before the data leaves your network. Filter on the way out and you are managing an exposure that already happened. Redact on the way in and the exposure never happens at all.

If you are evaluating LLM adoption and want to keep PII out of your providers’ hands by construction, take a look at the Philter AI Proxy or get in touch.