Policy-to-Eval Harness
Can a written usage policy be turned into something you can actually measure?
The missing step
Every frontier lab publishes a usage policy. Every lab enforces it with models. Almost nobody publishes the mapping between the two — the step where "do not provide material assistance toward unauthorized intrusion" becomes a decision an evaluator can score.
This builds that step explicitly, then measures how far model behaviour sits from the policy as written.
It was designed to measure the direction usually left unmeasured — over-refusal, where a model declines a legitimate request — and it reports leakage in the same table, because either number alone is misleading.
The finding
Leakage beat over-refusal by 1.8×.
14.2% of policy-prohibited requests got a substantive answer, against 7.7% of policy-permitted requests refused. That inverts the framing this project was built around.
These models are not paying for safety with over-refusal; they are permissive in both directions, and a summary quoting only the over-refusal figure would call them well-calibrated, which they are not.
The models sit on a clean permissiveness frontier, not a safety ranking:
| Model | Over-refusal ↓ | Leakage ↓ | Recall on prohibited |
|---|---|---|---|
| mistral-nemo-12b | 1.3% | 47.4% | 0.53 |
| qwen2.5-14b | 3.3% | 14.4% | 0.86 |
| llama3.1-8b | 10.2% | 5.2% | 0.95 |
| llama3.2-3b | 11.6% | 0.0% | 1.00 |
| gemma2-9b | 12.2% | 4.1% | 0.96 |
mistral-nemo-12b answered nearly half of all prohibited requests —
including one asking how to increase a pathogen's transmissibility.
llama3.2-3b refused every prohibited request and 11.6% of legitimate
ones. Same prompts, same policy, opposite failure modes.
There is no single number that separates these models usefully, which is the point. This does not produce a safety score. "Lower over-refusal" is not "better", as the table makes obvious.
Where the measurement itself broke down
The judge was calibrated against 112 hand-labelled rows. Cohen's κ = 0.52 unweighted, 0.68 linear-weighted — and the shape of the disagreement is the more useful result.
- 100% of disagreements ran one direction — the judge crediting less delivered substance than the human — and 97% were one notch apart.
- Random rater noise splits evenly and scatters. This is a calibration offset, not noise.
- The cause is an ambiguity in the rubric, not an error by either rater. On responses that decline the specific ask but deliver substantive general content — the most common shape in the data — two rubric rules both fire and contradict each other.
Reporting the rubric defect alongside the results is the part I care most about. An eval that hides its own calibration failure is worse than no eval, because it launders a measurement problem into a model finding.
Responsible use
The repository ships a responsible-use policy and a disclosure directory. The borderline prompt set is built to probe policy boundaries, and it is handled accordingly — which is why this repository was released deliberately rather than quickly, after a disclosure review of the prompt set.
Next project
A discrete-event simulation of a Trust & Safety enforcement queue — and why the aggregate SLA number is structurally incapable of showing you a failure.
Enforcement Operations Simulator