Alerting on PII Spikes with Phield and Slack

This guide sets up Phield to post to a Slack channel when the amount of PII coming out of one of your systems jumps above its usual level. You report how many of each PII type a job or service saw, Phield keeps a baseline for each one, and a count well above that baseline becomes an alert.

It is useful wherever a sudden rise in PII volume means something went wrong: an upstream field that started carrying many more card numbers, a logging change that writes far more email addresses than before, or a redaction step that stopped running. The alert compares each count with that series’ own average, so a PII type has to have been seen before it can spike. A type that appears for the first time, with an average of zero, does not trigger the default percentage alert. Phield only sees the counts you send it, so its alerts are as good as the detection that produced those counts. Detection is probabilistic. Check the counts against a sample of your own data before you rely on them.

1. Start Phield with storage, a key, and a webhook

Phield keeps its data in memory unless you give it MongoDB, and in-memory baselines are lost on restart. For alerting you want them to survive, so point it at a MongoDB database. Set an API key so only your pipeline can write counts, and give it a Slack Incoming Webhook URL.

docker run -d -p 8443:8443 \
  -e PHIELD_MONGO_URI=mongodb://mongodb.internal:27017/phield \
  -e PHIELD_API_KEY=replace-with-a-long-random-key \
  -e PHIELD_SLACK_WEBHOOK_URL=https://hooks.slack.com/services/T000/B000/XXXX \
  -e PHIELD_ALERT_THRESHOLD=0.2 \
  -e PHIELD_WINDOW_SIZE=24 \
  philterd/phield

The database name comes from the URI. The last two variables are the defaults, written out so you know which knobs you will tune later. With the default percentage_delta method, a count breaches when it is more than PHIELD_ALERT_THRESHOLD above the average for that series over the last PHIELD_WINDOW_SIZE hours. A threshold of 0.2 means 20 percent.

The container serves HTTPS on port 8443 with a certificate it generates when it starts, so curl needs -k until you mount your own and set PHIELD_CERT_FILE and PHIELD_KEY_FILE. Check that it is up:

curl -k https://localhost:8443/health

{"applicationVersion":"1.0.0","status":"UP"}

/health returns 503 with a status of DOWN when Phield cannot reach MongoDB, so you can point a load balancer or probe at it. It needs no key.

2. Send counts from your pipeline

After each batch or on a fixed interval, post the counts to /ingest. The counts are numbers per PII type, never the values themselves.

curl -k -X POST https://localhost:8443/ingest \
  -H "Authorization: Bearer $PHIELD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "source_id": "claims-intake-worker",
        "organization": "example-corp",
        "context": "production",
        "pii_types": {"email": 118, "phone-number": 42, "ssn": 3}
      }'

Phield answers 202 with {"status":"accepted"}. A few things decide how useful the alerts are:

• Each combination of source_id, organization, context, and PII type is its own series with its own baseline. Keep source_id stable for a given system, and use context to separate things that should not share a baseline, such as production and staging. Set context explicitly. organization falls back to default when omitted.
• Send on a regular cadence so each data point covers a comparable amount of work. A batch that is ten times larger than usual will look like a spike.
• timestamp is optional and defaults to the time Phield receives the count. The average is taken over the window ending now, so backfilling old counts does not build a baseline for live alerting.
• Phield rejects a request with 400 and an error message when source_id is blank, pii_types is empty, a type name is blank, contains a ., or starts with $, or a count is negative. Without the bearer token, or with the wrong one, it returns 401.

If your counts already go through Kafka, set PHIELD_KAFKA_BROKERS (and optionally PHIELD_KAFKA_TOPIC, default phield-pii-counts) and publish the same JSON body to the topic instead. Use the field names shown above. The API key does not apply to Kafka, so control writes with Kafka’s own authentication.

3. Read the alerts

When a count breaches, Phield writes a line to standard output, records the breach, and sends the same text to Slack:

[TREND BREACH] Source: claims-intake-worker, Organization: example-corp, Context: production, PII Type: ssn, Current: 27, Avg: 4.10, Increase: 558.54%

Avg is the window average the count was compared against, and Increase is how far above it the count was. To check a breach, look at what changed in that source around that time: a deploy, a new data feed, a schema change, or a redaction step that was skipped.

After an alert, the same series stays quiet for PHIELD_COOLDOWN_MINUTES (default 60). Breaches during the cooldown are logged with [SUPPRESSED] and are not sent to Slack or recorded. The cooldown clears early once the series has three readings in a row back near its baseline, so a new spike after recovery alerts right away.

The dashboard at https://localhost:8443/dashboard shows the alert timeline next to the volume for each PII type. To pull recent breaches into another tool, use the JSON endpoint behind it, which takes hours up to 168:

curl -k "https://localhost:8443/api/dashboard/alerts?hours=24"

The dashboard and its endpoints are not covered by the API key. They return aggregate counts only, but restrict them at the network level, or set PHIELD_DASHBOARD_ENABLED=false if nobody needs them.

4. Tune the threshold with replay

Once a few days of counts are stored, test a different threshold against them before changing the live setting. /replay reruns detection over a time range and reports what would have fired. It sends nothing to Slack and does not change the stored baselines.

curl -k -X POST https://localhost:8443/replay \
  -H "Authorization: Bearer $PHIELD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
        "start_time": "2026-11-05T00:00:00Z",
        "end_time": "2026-11-12T00:00:00Z",
        "test_threshold": 0.5,
        "pii_types": ["ssn", "email"]
      }'

The response has total_points_processed, virtual_breaches_detected, and a breach_details list with the timestamp, pii_type, source_id, count, and average of each one. If the list is cut short, breach_details_truncated is true. Compare the breaches against the incidents you know about, raise the threshold if the noisy ones dominate, and lower it if a real event is missing. Then set PHIELD_ALERT_THRESHOLD and restart.

Replays are limited to 90 days per request and one at a time by default. A second concurrent replay gets 429.

5. Mute a context during planned changes

When you expect a jump, such as a migration that loads historical records, mute the context instead of ignoring the channel:

curl -k -X POST https://localhost:8443/mute \
  -H "Authorization: Bearer $PHIELD_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"organization": "example-corp", "context": "production", "minutes": 120}'

A mute stops notifications for that organization and context only. Counts still update the baselines, and breaches are still recorded and marked [MUTED] in the log, so the dashboard shows what happened while the channel was quiet.

Next steps

If percentage changes are too noisy for a series that swings a lot, look at the z_score trend method in the documentation, which learns each series’ normal variation and alerts on standard deviations instead. To page someone rather than post to a channel, set PHIELD_PAGERDUTY_ROUTING_KEY. Slack and PagerDuty can run together.