Integrating Philter AI Proxy with OpenAI and Anthropic in Production

Putting Philter AI Proxy in front of OpenAI or Anthropic takes one line of configuration. The proxy speaks each provider’s native wire protocol, so you change a base URL and your existing SDK keeps working. That part takes about a minute.

The parts that decide whether the deployment actually holds are the three nobody asks about first: where the proxy sits relative to the AI gateway you probably already run, what happens to a streaming response, and which problems the proxy has deliberately decided are not its job. All three changed recently, so if you last looked at this a few months ago some of it will be new.

The one-line change

For OpenAI, point the base URL at the proxy. Nothing else moves, including the API key, which the proxy forwards.

from openai import OpenAI

client = OpenAI(base_url="https://philter-ai-proxy.internal:8080/v1")

For Anthropic, the same idea. The proxy serves /v1/messages, and the usual headers pass straight through.

curl https://philter-ai-proxy.internal:8080/v1/messages \
  -H "Content-Type: application/json" \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -d '{
    "model": "claude-sonnet-5",
    "max_tokens": 1024,
    "messages": [{"role": "user", "content": "Summarize this patient note: ..."}]
  }'

That is genuinely the whole client-side integration for both providers. Gemini, Vertex AI, Bedrock, Azure OpenAI, Ollama, and any OpenAI-compatible backend work the same way, each on its own native path.

Where it goes matters more than how you configure it

Most teams running LLM traffic in production already have an AI gateway doing routing, failover, rate limiting, and caching. The proxy is not a replacement for that, and the ordering of the two is the single decision worth thinking about.

Put the gateway in front and the proxy behind it, so redaction is the last thing that happens before data leaves your network. The reason is not aesthetic. If the gateway sits after the proxy, its own features become bypass routes: a fallback to a provider the proxy never saw, or a system prompt the gateway injects after redaction has already run, puts unredacted text on the wire. Neither is a bug in the gateway. It is doing exactly what you configured it to do, on traffic the redaction step no longer controls.

With the proxy last, no plugin, retry path, or model-fallback rule can get around it, because there is nothing after it but the provider.

On the gateway side this is one line too. In LiteLLM, point api_base at the proxy:

model_list:
  - model_name: gpt-4
    litellm_params:
      model: openai/gpt-4
      api_base: https://philter-ai-proxy.internal:8080/v1
      api_key: os.environ/OPENAI_API_KEY

Portkey takes a custom_host on the virtual key, and Kong’s ai-proxy plugin takes an upstream override. The proxy needs no matching configuration for any of them, since it is speaking the provider’s protocol either way.

What it deliberately does not do

Recent releases removed a set of features rather than adding them, and if you have a config from earlier this year it is worth knowing which.

Rate limiting is gone, along with its Redis backend and its 429 response. The response cache is gone, including the X-Cache header and its metrics. Per-key concurrency caps are gone. Token quotas, spend tracking, and usage export were never a good fit and have been removed as well.

All of that belongs to an AI gateway, which is already sitting in front and already doing it. Two components implementing the same feature is how you get a cache that keys per tenant in one place and not the other, which was a real cross-tenant risk in the proxy’s implementation.

Two practical consequences. First, dropping the cache and the rate limiter removed the last Redis dependency, so the proxy now keeps no shared state at all. Horizontal scaling is running more copies. Second, if your config still sets any of the removed keys, they are ignored rather than rejected, and startup logs a warning naming each one. No config version bump, no migration, but do read those warnings rather than assuming a limit you configured is still being enforced somewhere.

What stays is overload protection that does not need shared state: listen.maxConcurrentRequests for in-flight requests, and a ceiling on simultaneous TLS handshakes so a connection flood cannot spawn unbounded goroutines.

Streaming, and the failure mode that was hiding in it

This is the part to read carefully if you scan responses on the way back.

Outbound scanning has three actions: redact, flag, and block. A streaming response cannot be scanned the way a complete one can, because there is no complete body to send to Philter until the stream is over. The proxy used to forward those streams unscanned.

The problem is who chooses. Streaming is a client-side flag, so any caller could opt out of your outbound policy by setting "stream": true. A configuration that read as “block responses containing PII” was, in practice, “block responses containing PII unless the caller would rather not.”

Under outbound.action: block, the proxy now rejects an unscannable streaming response with a 403 carrying pii_blocked and outbound_stream_unscannable, instead of forwarding it. That is a behavior change on existing block routes, and it is the kind that shows up as user-visible errors rather than as a log line, so roll it out somewhere you can watch. If you need the old behavior, it is still available:

defaults:
  outbound:
    enabled: true
    action: block
    allowUnscannedStreams: true   # forward unscannable streams instead of rejecting

Setting that flag is a real choice with a real cost, which is the point of making it explicit. The redact and flag actions are unchanged.

Streaming improved in a less alarming way too: token usage is now accounted for streaming responses on the OpenAI and Anthropic paths, pulled from the final usage event without buffering the stream. The same audit-log fields and Prometheus counters you get from non-streaming responses now populate for streaming ones, which matters if you were reconciling proxy metrics against a provider bill and finding a gap exactly the size of your streaming traffic. Bedrock’s ConverseStream is supported on the same terms, forwarded incrementally rather than buffered.

Policy per route

One more thing worth setting up on day one. Policies attach per route, matched on path and model, so a healthcare chatbot and an internal analytics tool can share a proxy without sharing a redaction policy.

routes:
  - match:
      path: /v1/messages
    policy: hipaa-safe-harbor
    context: healthcare-chatbot
    outbound:
      enabled: true
      action: block

  - match:
      path: /v1/chat/completions
      model: gpt-4
    policy: general-purpose
    context: internal-analytics

The context value flows into the audit log, which is what makes the log answer questions later. Without it you can see that something was redacted. With it you can see which application it came from.

Before you call it done

Detection is probabilistic, and a proxy is a control point rather than a guarantee. The deployment is finished when you have measured the policy against your own documents, not when traffic is flowing. Philter Scope scores a policy on precision and recall against a gold standard, and doing that before the system carries real data is considerably easier than doing it afterward.

Two smaller things to confirm on the way: that the audit log is going somewhere you will actually read, and that you have tried a request with "stream": true against every route where you set action: block. The second one takes thirty seconds and tells you whether your outbound policy means what you think it means.