Measure Philter's Redaction Accuracy on Your Own Documents

This guide walks through scoring a Philter policy against a small set of your own documents with Philter Scope. You label the PII in a sample of documents, run philterscope-audit against a running Philter, and get precision, recall, and F1 overall and per entity type. Do this before you put a policy into production, and again after every change to it, so you know what the change did on your data rather than on someone else’s benchmark.

For why these three numbers matter and how targets differ by industry, see Evaluating redaction with Philter Scope. This guide is the procedure.

You need:

• A running Philter 4 and an API key with the redact scope.
• The philterscope-audit and philterscope-serve binaries. Build them from the repository with make build, or use the Docker image described in the documentation.

1. Label a sample of your documents

Pick documents that look like what the policy will see in production. Copy each one into a golden/ directory and wrap every piece of PII in a tag named for its type:

Patient <NAME>Jane Roe</NAME> called on <DATE>03/14/2026</DATE> from <PHONE_NUMBER>555-0142</PHONE_NUMBER>.
Callback requested at <EMAIL>jane.roe@example.com</EMAIL>.

Tag names must be uppercase letters and underscores. Anything else, such as <Name> or <PHONE-NUMBER>, is not read as a tag and stays in the text. The names are yours to choose. They do not have to match Philter’s filter names, because matching is done on character positions, not labels.

Label carefully. Every span you miss in the gold standard turns a correct redaction into a false positive, and every span you mislabel skews the per-entity numbers.

2. Create the raw copies

The text sent to Philter must be the golden text with the tags removed, character for character, because the golden spans are offsets into that text. Generate the raw files from the golden ones rather than keeping two copies by hand:

mkdir -p audit/golden audit/raw
# put your tagged files in audit/golden/, then:
for f in audit/golden/*.txt; do
  sed -E 's#</?[A-Z_]+>##g' "$f" > "audit/raw/$(basename "$f")"
done

Each raw file is matched to the golden file with the same name. Keep only documents in raw/, since every file in the input directory is audited.

3. Run the audit

Point the audit at Philter, the policy you want to score, and the two directories:

./philterscope-audit \
  --url https://localhost:8080 \
  --token "$PHILTER_API_KEY" \
  --ca-cert philter.pem \
  --policy default \
  --input ./audit/raw \
  --golden ./audit/golden \
  --output ./audit/report \
  --threshold 0.9

Philter’s Docker image serves HTTPS with a self-signed certificate, so --ca-cert names a PEM file to trust. You can save it with:

openssl s_client -connect localhost:8080 </dev/null 2>/dev/null | openssl x509 > philter.pem

Each raw file is posted to Philter’s /api/explain endpoint with the named policy, and the returned spans are compared with your labels. The audit prints one line, Audit complete. Precision: ..., Recall: ..., F1: ..., and writes report.html and report.json to the output directory. It also saves the result to audit history, in a .philterscope directory under the current directory, or in MongoDB if PHILTERSCOPE_MONGODB_CONNECTION_STRING is set.

The run exits with status 1 if Philter cannot be reached, if Philter refuses the key, or if no file could be scored. A single file that fails is skipped and listed under skipped in report.json. Skipped files contribute nothing to the metrics. A run that scores below your thresholds still exits with status 0, so a CI job that should fail on low recall reads report.json. Every entity type whose recall is below its target gets a recommendation of kind recall_below_threshold, so this step fails the build when there is one:

jq -e '[(.recommendations // [])[] | select(.kind == "recall_below_threshold")] | length == 0' \
  ./audit/report/report.json

4. Read the results

Matching is by position. A labeled span that any Philter span overlaps, exactly or partially, is a true positive. A labeled span with no overlapping detection is a false negative. A detection that overlaps no labeled span is a false positive. The counts are summed across all documents, and then:

• Precision is TP / (TP + FP): the share of detections that hit labeled PII.
• Recall is TP / (TP + FN): the share of labeled PII that Philter found.
• F1 is the harmonic mean of the two.

Partial overlaps count as found. A detection that covers “Jane” but not “Roe” scores the same as one that covers the whole name, even though part of the name would remain in the output. Check the per-document results, where each match is marked EXACT or PARTIAL, before you trust a high recall number.

Per entity, true positives and false negatives are credited to your golden label, and false positives to the label Philter returned. The confusion matrix shows the rest: a (missed) column for labeled spans nothing detected, a (none) row for detections with no label, and off-diagonal cells where Philter found the span under a different type.

--threshold sets the recall target for every entity, and --thresholds "NAME=0.95,SSN=1.0" overrides it per entity. The keys must match your golden tag names exactly. Each entity is marked PASS or FAIL against its recall target. Precision is not scored against a target. An entity whose precision falls below --precision-floor (default 0.25) is marked CHECK, since that usually means a filter is matching far more than intended.

To browse the report:

./philterscope-serve --report ./audit/report/report.json --threshold 0.9 --privacy

Pass the same --threshold and --thresholds you audited with. When serving a single report, philterscope-serve replaces the report’s targets with its own flags, and --threshold defaults to 0.5. --privacy replaces the text of each span in the UI with a short hash, though the report files still contain the original text.

What to do next

report.json includes a recommendations list: suggested policy changes for each entity below its recall target, and a warning for each entity below the precision floor. They are advisory. Philter Scope does not edit your policy. The recommendations reference describes each field.

Make one change to the policy, re-run the same command against the same golden set, and compare. Use --group to give related runs a shared name in the history. The numbers describe your sample and nothing beyond it. A policy that scores well here can still miss PII in documents that look different, so grow the golden set when new kinds of documents arrive, and keep checking output on your own data.