A false-positive rate you can prove

Alert fatigue is the real cost of detection. Every vendor promises to reduce false positives; almost none can put a number on it before you deploy, and the numbers they do quote are averages measured once, on a benchmark that is not your network.

Ward³ takes a different route. Instead of chasing false positives with thresholds, it bounds them by construction — and the difference between those two ideas is the whole point of this post.

Empirical vs. guaranteed

An empirical claim says: on this dataset, with this tuning, we saw this false-positive rate. It is real, but it is a measurement of one past run. Change the traffic mix — a new tenant, a seasonal shift, a different protocol split — and the number moves, silently.

A guarantee says: alert under this rule and the false-positive rate cannot exceed α, as a property of the method, on any traffic. That is what conformal prediction gives you, and it is a stronger thing to be able to say.

The conformal p-value

Each judge's raw score is turned into a p-value against a per-tenant baseline of benign traffic, using a split-conformal upper-tail estimate:

• p = (1 + #{ benign ≥ score }) / (n + 1)
• Alert when p ≤ α, and the false-positive rate is bounded by α.
• The guarantee is distribution-free and finite-sample — no assumption about the shape of benign traffic, and it holds for the n samples you actually have, not only in the limit.

Per-tenant, per-judge, and one dial

Calibration is stratified by (tenant, judge, model version), so a noisy tenant never inflates a quiet one, and every deployment gets a bound tuned to its own normal. Tightening the whole platform is a change to α, not a training run: war mode simply lowers it.

And it fails closed. A brand-new model or an empty calibration window yields p = 1 — the judge abstains instead of guessing. Robustness has to hold on day one, not only at steady state.

What it does not do

Conformal calibration bounds false positives; it does not invent separability. If a judge genuinely cannot tell an attack from benign traffic, a bounded p-value will faithfully report that it can't. That honesty is a feature: it turns a noisy judge into one you can use at a known cost, and it stops the mediator from consuming a score that has no calibrated meaning.