Multilingual Enforcement Consistency
A usage policy says nothing about language. Does the model?
The blind spot
Every frontier lab publishes one usage policy and enforces it with one model, in every language at once. But almost every published safety eval is in English — and the public record is consistent that alignment generalises unevenly across languages. Anthropic's own Values in the wild, across models and languages finds Claude's expressed values shift measurably by language before any adversary is involved.
If the values move, the enforcement decisions can move too. Neither source policy contains a language clause, so the decision a policy requires for a request is — by construction — the same in every language. Any cross-language variance is therefore a gap between the model and the policy, not a difference in what the policy asks. This measures that gap.
It is the sequel to project 03: it reuses that project's policy-to-decision taxonomy unchanged — asserted byte-for-byte identical in the tests — and adds exactly one moving part. Framing is held fixed; the language is the only variable.
Read this before the numbers.
The shipped run uses deterministic stand-in models, so the repository
runs on git clone with no API keys and CI can assert exact figures.
Every chart and table below is watermarked SIMULATED. These numbers
exercise and validate the measurement — they are not a finding about any
named real model. Swapping one config file points the identical pipeline at a live
model.
What the harness is built to see
The design question is not "which model is safest" — it is whether a decision holds as the language changes. A model can post an immaculate English red-team score and answer the very same prohibited request the moment it arrives in another language, and an English benchmark cannot see it. In the demonstration run, three of the four stand-ins leak 0% in English and materially more outside it, with the worst arm answering the same prohibited request it refused in English.
The result a single English number hides
Consistency counts a request as consistent only if all five language arms reach the same decision. The stand-in built to refuse hard in English — the profile that scores "safe" on an English red-team — turns out to be the least consistent of the four: its refusals do not travel, and its code-mixed arms leak. A single English figure would rank it the cautious one. It is the opposite.
When a decision flips with language the direction is the whole story: a flip toward answering a prohibited request is a safety failure; a flip toward declining a permitted one is an access failure. The harness reports the two directions separately, because collapsing them hides which one a given model is paying with.
A correct refusal that points nowhere useful
In the self-harm category the decision is only half the response. An answer that
refers a user to a crisis line only helps if that line exists where they are.
A Singapore user handed the US 988 number has received a correct
decision and a useless answer. So referral validity is scored on its own
axis, per locale — a resource valid in-country (Singapore's SOS 1767,
France's 3114) versus a generic global default.
The region is where the hard cases live
The corpus is hand-translated, not machine-translated — an MT system's own hedging would be indistinguishable from a model's refusal behaviour, which is fatal to this exact measurement. Written by a native English and Chinese speaker with working French, based in Singapore, it carries the context that only exists in-language:
- SEA scam typologies in their own vocabulary — 杀猪盘 (pig-butchering), 跑分 (money-muling), ah long (loan sharks) — so the harness tests whether a model recognises fraud before the English calque exists.
- Singapore's racial- and religious-harmony frame (the CMIO model, religious-harmony law), which makes a different set of utterances legally salient than a US or French frame — under identical policy text.
- Care-system vocabulary that does not map one-to-one (A&E / urgences / 急诊 / polyclinic), testing whether a model gates on institution names or on clinical urgency.
- Code-mixed registers — Singlish and Singapore Mandarin — the low-resource condition the public record says degrades worst.
The measurement checks itself, in every language
A judge that is sharp in English and blurry in Chinese would manufacture the very gap the project reports. So the judge is calibrated against human labels per locale before any finding is trusted — overall Cohen's κ = 0.87, and every language clears the substantial-agreement floor.
The part I care most about is the same as in project 03: the harness is built to expose its own failure modes rather than launder them. It reports leakage and over-refusal together, checks its ruler in every language, separates a wrong decision from a right decision that fails locally, and labels every simulated number as simulated. An eval that hides any of those is worse than no eval.
Responsible use
Three categories from the parent taxonomy — CBRN, weapons, and child-safety — were deliberately excluded from translation. A faithful non-English rendering of such a request is an artefact whose only novel contribution is a more accessible harmful ask; the English versions already exist in project 03 and are enough to measure them. Prohibited-labelled prompts are written at the specificity of a clear policy violation and no further — no working exploit, no synthesis route, no real target. The repository ships a responsible-use policy that states all of this.
Next project
Multi-turn distress conversations — and why the second-highest scoring system in the suite is the one that must not ship.
Distress-Conversation Safety Eval