---
title: "Measure Philter's Redaction Accuracy on Your Own Documents"
description: "Measure PII redaction precision and recall on your own documents with Philter Scope: label a gold-standard set, audit a Philter policy, read the report."
url: "https://philterd.ai/guides/measure-redaction-accuracy-with-philter-scope/"
---

[Knowledge base](https://philterd.ai/guides/) [How-to](https://philterd.ai/guides/?kind=how-to)

# Measure Philter's Redaction Accuracy on Your Own Documents

Measure PII redaction precision and recall on your own documents with Philter Scope: label a gold-standard set, audit a Philter policy, read the report.

For [Philter Scope](https://philterd.ai/support/philter-scope/)

This guide walks through scoring a [Philter](https://philterd.ai/philter/) policy against a small set of your own documents with [Philter Scope](https://philterd.ai/philter-scope/). You label the PII in a sample of documents, run `philterscope-audit` against a running Philter, and get precision, recall, and F1 overall and per entity type. Do this before you put a policy into production, and again after every change to it, so you know what the change did on your data rather than on someone else’s benchmark.

For why these three numbers matter and how targets differ by industry, see [Evaluating redaction with Philter Scope](https://philterd.ai/blog/evaluating-redaction-with-philter-scope/). This guide is the procedure.

You need:

- A running Philter 4 and an API key with the `redact` scope.
- The `philterscope-audit` and `philterscope-serve` binaries. Build them from the [repository](https://github.com/philterd/philter-scope) with `make build`, or use the Docker image described in the [documentation](https://philterd.github.io/philter-scope/running/).

## 1\. Label a sample of your documents

Pick documents that look like what the policy will see in production. Copy each one into a `golden/` directory and wrap every piece of PII in a tag named for its type:

```text
Patient <NAME>Jane Roe</NAME> called on <DATE>03/14/2026</DATE> from <PHONE_NUMBER>555-0142</PHONE_NUMBER>.
Callback requested at <EMAIL>jane.roe@example.com</EMAIL>.
```

Tag names must be uppercase letters and underscores. Anything else, such as `<Name>` or `<PHONE-NUMBER>`, is not read as a tag and stays in the text. The names are yours to choose. They do not have to match Philter’s filter names, because matching is done on character positions, not labels.

Label carefully. Every span you miss in the gold standard turns a correct redaction into a false positive, and every span you mislabel skews the per-entity numbers.

## 2\. Create the raw copies

The text sent to Philter must be the golden text with the tags removed, character for character, because the golden spans are offsets into that text. Generate the raw files from the golden ones rather than keeping two copies by hand:

```bash
mkdir -p audit/golden audit/raw
# put your tagged files in audit/golden/, then:
for f in audit/golden/*.txt; do
  sed -E 's#</?[A-Z_]+>##g' "$f" > "audit/raw/$(basename "$f")"
done
```

Each raw file is matched to the golden file with the same name. Keep only documents in `raw/`, since every file in the input directory is audited.

## 3\. Run the audit

Point the audit at Philter, the policy you want to score, and the two directories:

```bash
./philterscope-audit \
  --url https://localhost:8080 \
  --token "$PHILTER_API_KEY" \
  --ca-cert philter.pem \
  --policy default \
  --input ./audit/raw \
  --golden ./audit/golden \
  --output ./audit/report \
  --threshold 0.9
```

Philter’s Docker image serves HTTPS with a self-signed certificate, so `--ca-cert` names a PEM file to trust. You can save it with:

```bash
openssl s_client -connect localhost:8080 </dev/null 2>/dev/null | openssl x509 > philter.pem
```

Each raw file is posted to Philter’s `/api/explain` endpoint with the named policy, and the returned spans are compared with your labels. The audit prints one line, `Audit complete. Precision: ..., Recall: ..., F1: ...`, and writes `report.html` and `report.json` to the output directory. It also saves the result to audit history, in a `.philterscope` directory under the current directory, or in MongoDB if `PHILTERSCOPE_MONGODB_CONNECTION_STRING` is set.

The run exits with status 1 if Philter cannot be reached, if Philter refuses the key, or if no file could be scored. A single file that fails is skipped and listed under `skipped` in `report.json`. Skipped files contribute nothing to the metrics. A run that scores below your thresholds still exits with status 0, so a CI job that should fail on low recall reads `report.json`. Every entity type whose recall is below its target gets a recommendation of kind `recall_below_threshold`, so this step fails the build when there is one:

```bash
jq -e '[(.recommendations // [])[] | select(.kind == "recall_below_threshold")] | length == 0' \
  ./audit/report/report.json
```

## 4\. Read the results

Matching is by position. A labeled span that any Philter span overlaps, exactly or partially, is a true positive. A labeled span with no overlapping detection is a false negative. A detection that overlaps no labeled span is a false positive. The counts are summed across all documents, and then:

- Precision is TP / (TP + FP): the share of detections that hit labeled PII.
- Recall is TP / (TP + FN): the share of labeled PII that Philter found.
- F1 is the harmonic mean of the two.

Partial overlaps count as found. A detection that covers “Jane” but not “Roe” scores the same as one that covers the whole name, even though part of the name would remain in the output. Check the per-document results, where each match is marked `EXACT` or `PARTIAL`, before you trust a high recall number.

Per entity, true positives and false negatives are credited to your golden label, and false positives to the label Philter returned. The confusion matrix shows the rest: a `(missed)` column for labeled spans nothing detected, a `(none)` row for detections with no label, and off-diagonal cells where Philter found the span under a different type.

`--threshold` sets the recall target for every entity, and `--thresholds "NAME=0.95,SSN=1.0"` overrides it per entity. The keys must match your golden tag names exactly. Each entity is marked PASS or FAIL against its recall target. Precision is not scored against a target. An entity whose precision falls below `--precision-floor` (default 0.25) is marked CHECK, since that usually means a filter is matching far more than intended.

To browse the report:

```bash
./philterscope-serve --report ./audit/report/report.json --threshold 0.9 --privacy
```

Pass the same `--threshold` and `--thresholds` you audited with. When serving a single report, `philterscope-serve` replaces the report’s targets with its own flags, and `--threshold` defaults to 0.5. `--privacy` replaces the text of each span in the UI with a short hash, though the report files still contain the original text.

## What to do next

`report.json` includes a `recommendations` list: suggested policy changes for each entity below its recall target, and a warning for each entity below the precision floor. They are advisory. Philter Scope does not edit your policy. The [recommendations reference](https://philterd.github.io/philter-scope/recommendations/) describes each field.

Make one change to the policy, re-run the same command against the same golden set, and compare. Use `--group` to give related runs a shared name in the history. The numbers describe your sample and nothing beyond it. A policy that scores well here can still miss PII in documents that look different, so grow the golden set when new kinds of documents arrive, and keep checking output on your own data.

## Related reading

- [Philter Scope: Scores a Philter policy's redaction against labeled test data.](https://philterd.ai/philter-scope/)
- [Philter Scope support: Guides, documentation, and the issue tracker.](https://philterd.ai/support/philter-scope/)
- [Philter Scope documentation: Every flag, the data formats, and the report fields.](https://philterd.github.io/philter-scope/)