Redact PII before it reaches the LLM

Philter AI Proxy

Philter AI Proxy is an airlock for your large language model (LLM) traffic: prompts pass through to the model after Philter redacts the sensitive data it detects. Prompts get redacted before they leave your network, and responses can be scanned on the way back.

View on GitHub

Star 7

Change one URL. Stop leaking PII to LLM providers.

# Before
client = OpenAI(api_key=KEY)

# After: just point at the proxy
client = OpenAI(
    api_key=KEY,
    base_url="https://philter-proxy.internal/v1"
)
# Same OpenAI client. PII redacted automatically.

Point your client's base_url at the running proxy and it redacts each prompt before forwarding it to your provider.

Documentation → Release Notes → GitHub →

What the LLM provider actually sees

Prompts routinely carry names, emails, and account numbers. Philter AI Proxy redacts what Philter detects before the request leaves your network, so the provider receives the redacted text instead. Detection is probabilistic, so validate the results against your own prompts.

Your application sends

Summarize this ticket: customer Margaret CollinsPERSON (mcollins@example.comEMAIL) reported that card 4111 1111 1111 1111CREDIT_CARD was charged twice.

The LLM provider receives

Summarize this ticket: customer {{REDACTED-PERSON}} ({{REDACTED-EMAIL}}) reported that card ****-****-****-1111 was charged twice.

Responses are scanned on the way back, too. Your code changes by one URL, nothing else.

Run it with Docker

The proxy ships as a multi-arch image on Docker Hub at philterd/philter-ai-proxy. Pull it, point a config file at your Philter instance and chosen provider, and run.

# 1. Pull the image from Docker Hub
docker pull philterd/philter-ai-proxy

# 2. Grab a starting config and edit it for your Philter and provider
curl -O https://raw.githubusercontent.com/philterd/philter-ai-proxy/main/config.example.yaml
mv config.example.yaml config.yaml

# 3. Run it: expose port 8080 and mount your config
docker run -p 8080:8080 \
  -v "$(pwd)/config.yaml:/app/config.yaml:ro" \
  -e PHILTER_PROXY_CONFIG=/app/config.yaml \
  philterd/philter-ai-proxy

Prefer Compose? The repo ships a docker-compose.yaml: clone it and run docker compose up. Every configuration option is documented in the installation guide.

The LLM privacy layer your stack is missing

Drop-in for major providers

Speaks the OpenAI, Azure OpenAI, Anthropic, Gemini, Vertex AI, Ollama, and Amazon Bedrock APIs, plus any OpenAI-compatible provider such as Mistral, Cohere, vLLM, or LM Studio. Your existing SDKs don't need a single line changed; only the base URL.

Inbound prompt redaction

Strips PII and PHI from prompts before they're forwarded to the model. Names, SSNs, MRNs, account numbers: all replaced according to your policy.

Authentication and access control

API key authentication and mutual TLS (mTLS) are both supported, independently or together. Per-key authorization scopes and per-key policy binding let you grant different access levels to different clients without running separate proxy instances.

Audit log for compliance

Every redaction is logged with timestamp, entity type, and direction (in/out). The exact paper trail HIPAA and GDPR auditors expect for AI workloads.

Policy-driven

Uses the same Phileas policies as the rest of the Philterd toolkit. Define once; apply across redaction, discovery, monitoring, and now AI traffic.

Self-hosted

Runs inside your perimeter. The proxy is the last hop before LLM-bound traffic leaves your network, and the first hop on the way back: the airlock that sensitive data cannot cross.

Observability & operations

Production-ready from day one. The proxy exposes the signals your platform team needs to operate it with confidence.

Prometheus metrics

A /metrics endpoint exposes request counts, request and redaction latency, entities redacted by type, and Philter and provider error counts, labeled by provider. Drop it into your existing Grafana stack without custom instrumentation.

Structured audit logs

Every request is written as a JSONL record: entity types redacted, direction (inbound or outbound), model, policy, document ID, latency, client IP, and HTTP status. The exact audit trail HIPAA and SOC 2 reviewers expect for AI workloads.

Health endpoints

/livez and /readyz are the Kubernetes liveness and readiness probes, and /health reports Philter backend reachability as structured JSON for a load balancer or uptime monitor.

Frequently asked questions

If something here isn’t covered, get in touch and we’ll answer.

Which LLM providers does the proxy work with?

It speaks the OpenAI, Azure OpenAI, Anthropic Claude, Google Gemini, Vertex AI, Ollama, and Amazon Bedrock APIs, plus any OpenAI-compatible provider such as Mistral, Cohere, vLLM, or LM Studio. Switching providers is a configuration change, not a code change.

Do I have to change my application code?

No. Point your existing SDK at the proxy's base URL and everything else stays the same. The same OpenAI or Anthropic client keeps working; the proxy redacts each prompt and forwards it to your configured provider.

Does the proxy redact model responses too, or only prompts?

Both, but not the same way. Every prompt is redacted on the way out, before it leaves your network. Scanning responses on the way back is opt-in per route: when enabled, the proxy passes the reply through Philter and can redact, block, or flag any PII it finds. A streaming response cannot be scanned, so under the block action the proxy rejects it unless the route allows unscanned streams.

What does the proxy use to detect and redact PII?

The same Phileas policies as the rest of the Philterd toolkit, applied through your self-hosted Philter backend. Define a policy once and apply it across redaction, discovery, monitoring, and now AI traffic, with no per-surface drift.

How does the proxy map to SOC 2 and HIPAA controls?

Its features line up with specific SOC 2 Trust Services Criteria and HIPAA Security Rule safeguards: authentication and mTLS for access control, the audit log for audit controls, TLS everywhere for transmission security, and redaction itself for confidentiality and minimum-necessary handling. The SOC 2 and HIPAA control mapping guide lays out the full mapping and the configuration steps to support your own attestations.

Does the proxy log prompt or response content?

No. The audit log records metadata only: provider, model, the entity types redacted, counts, direction, latency, client IP, document ID, HTTP status, and a trace ID. Message content and filtered text are never written to the log. (Client IPs may be personal data under GDPR, so govern the log accordingly.)

How does the proxy fit with an AI gateway like LiteLLM or Portkey?

Philter AI Proxy is a redaction layer, not an AI gateway. It does redaction and the audit trail that goes with it, and it deliberately leaves provider routing, failover, rate limiting, response caching, token quotas, and per-tenant billing to a gateway such as LiteLLM, Portkey, or Kong AI Gateway. The proxy is built to run alongside one, on the last hop before traffic leaves your network, so whatever the gateway routes has already been redacted. The Using with an AI Gateway guide covers the deployment pattern.

Is Philter AI Proxy open source?

Yes. Philter AI Proxy is open source under the Apache License, version 2, and the code is on GitHub.

Ready to use Philter AI Proxy?

Three ways to get going: deploy the open source yourself, spin it up from a cloud marketplace, or work with our team directly. Pick the path that fits.