LLM Guardrail Benchmarks: Build Your Own Eval Set

Every guardrail vendor publishes an F1 score. None of them are measured on your traffic, against your threat model, at your tolerance for false positives. That is not an accusation of dishonesty — it is a structural property of benchmarks. A number produced on a public evaluation set describes how a detector behaves on that set’s distribution, and the distribution of prompts hitting a legal-document assistant looks nothing like the distribution hitting a customer-support bot or a coding agent.

The consequence is that selecting a guardrail on published numbers alone is guesswork, and the fix is not to distrust benchmarks but to build a small one of your own. This is a method for doing that: what the public sets measure, what an internal eval set needs to contain, which metrics decide the question, and how to keep the whole thing from rotting after the first quarter.

What the Public Benchmarks Actually Measure

Three reference points are worth understanding before building anything, because each one solves a different piece of the problem and the design decisions behind them are the ones you will have to make yourself.

PINT (Lakera) targets the leakage problem head-on. Its stated purpose is to evaluate prompt injection detection without relying on public datasets that a detector may have been trained on. The set holds 4,314 inputs — 3,016 English and 1,298 across more than 20 other languages — and the category mix is the instructive part: roughly 5.2% prompt injections, 0.9% jailbreaks, 20.9% deceptive benign inputs, and 36.5% each of ordinary user-agent chat and public document excerpts. Attacks are a small minority. Around a fifth of the corpus is benign text deliberately built to look like an attack.

JailbreakBench standardises the jailbreak side. Its JBB-Behaviors dataset holds 100 distinct misuse behaviours across ten categories aligned to OpenAI’s usage policies (55% original, the rest drawn from AdvBench, the Trojan Detection Challenge, and HarmBench), plus 100 benign behaviours included specifically to measure overrefusal. It ships a defined threat model, fixed system prompts and chat templates, scoring functions, and an evolving repository of adversarial prompt artifacts, so that two people reporting a number are reporting the same number.

HarmBench (arXiv:2402.04249) is the scale reference for the attack side, comparing 18 red teaming methods against 33 target LLMs and defenses in a single framework. Its contribution to your own work is mostly methodological: robustness is a property of a pair — this defence against this attack panel — not a property of the defence alone.

Vendor-neutral comparisons such as the TrueFoundry evaluation then show why single-number selection fails in practice. Across 400 category-balanced samples per task, the leaders were three different products: OpenAI Moderation on content moderation (F1 0.899), Azure PII on PII detection (F1 0.928, at 1.000 precision), and Pangea on prompt injection (F1 0.853, on a high-recall strategy). No provider won across all three. That result is discussed in more depth in the comparison of guardrail frameworks for 2026 and the evaluation of prompt injection detection tools.

Take three lessons from all of this: hard negatives dominate a realistic corpus, overrefusal needs its own labelled set, and a score is only interpretable next to the attack panel it was produced against.

Metrics That Decide the Question

F1 is the wrong headline metric for a production guardrail, because it assumes false positives and false negatives cost the same. They almost never do. A blocked legitimate customer request is a support ticket and a churn risk; a missed injection in an agent with write access is an incident. Pick the operating point from those costs, then report the following four numbers together.

False positive rate at fixed recall. Fix recall at the level your risk appetite demands — 0.90, 0.95, whatever the security owner signs off on — and then compare candidates purely on the FPR each one pays to get there. This is the number that determines whether the guardrail is deployable, and it is the one vendor material reports least often.

Attack success rate under an adaptive panel. ASR is the fraction of attack attempts that reach a policy-violating output despite the control. Run it against a panel that includes obfuscation, encoding, multi-turn build-up, and translation, not only the plain-text payloads. A detector can post a low ASR against the published attack set and a high ASR against a variant produced by anyone who tested against your stack first.

Latency at p95, not mean. A classifier averaging 12ms with a p95 of 180ms will show up as a user-visible stall on the exact requests that are longest and most likely to matter. Measure it at your real input lengths; the latency of a 200-token prompt tells you little about a 12k-token RAG context.

Cost per thousand requests. Includes the inference cost of the detector itself plus the escalation rate to any heavier second-stage check. A design that is cheap at 1% escalation becomes expensive at 15%, and the escalation rate is set by your traffic, not by the vendor.

Report all four in one table per candidate. Any single one of them can be gamed by moving a threshold; together they cannot.

Building the Eval Set

Aim for 800 to 1,500 labelled items. Below roughly 500 the confidence intervals swamp the differences between candidates; above a few thousand, labelling quality — not sample count — becomes the limiting factor.

Bucket 1 — real benign traffic (about 50%). Sampled from production logs, redacted, and stratified so that the long tail of unusual-but-legitimate requests is represented rather than averaged away. This bucket is what makes your FPR number mean anything. The field-level scheme in LLM audit logging is what makes sampling it possible without exporting raw user text.

Bucket 2 — hard negatives (about 20%). Benign prompts that resemble attacks: a security engineer asking how prompt injection works, a user quoting an error message full of instruction-like text, a document containing the phrase “ignore previous instructions” in a legitimate context. Follow PINT’s proportions here; this bucket is where naive keyword and regex controls collapse.

Bucket 3 — direct attacks (about 15%). Injection and jailbreak payloads relevant to your surface. A RAG application needs indirect injection embedded in retrieved documents; an agent needs payloads targeting tool invocation. Generic jailbreak corpora under-represent both.

Bucket 4 — obfuscated variants (about 10%). Each attack in Bucket 3, transformed: base64 and other encodings, homoglyphs, zero-width character insertion, translation into a lower-resource language, and split across multiple turns. Generate these mechanically so the transformation set is reproducible.

Bucket 5 — overrefusal probes (about 5%). Requests that sound alarming and are entirely legitimate for your domain — a medical assistant asked about drug interactions, a security tool asked to explain a CVE. Borrowed directly from JailbreakBench’s benign-behaviours design, and the fastest way to catch a guardrail that is “safe” because it refuses everything.

Two discipline rules make or break the whole exercise. Hold the set out. The moment anyone tunes a threshold or fine-tunes a classifier against these exact items, the number stops predicting field behaviour — which is the failure mode PINT was built to avoid. Keep a second, larger development set for tuning and touch the eval set only to produce final numbers. And label twice. Two independent labellers with a documented adjudication rule, because a corpus where the labels disagree with your own policy produces a benchmark that measures the labellers.

Running It: Placement, Not Just Product

Benchmark the deployment, not the detector in isolation. The same classifier placed before the model, after the model, or on both sides produces different numbers, and the choice interacts with your latency budget — the decision framework in choosing runtime guardrails for LLM apps covers that placement question directly.

Three configurations are usually worth measuring:

Single input classifier. The cheapest baseline; establishes what one control buys.
Input plus output classifier. Catches jailbreaks that pre-inference scanning missed, at the cost of inference already spent — the tradeoff described in jailbreak detection for LLMs.
Cascade. A fast, permissive first stage escalating a flagged minority to a heavier check. Report the escalation rate alongside the accuracy numbers, because it drives both cost and p95.

Wire the run into CI rather than a notebook. Adversarial testing harnesses such as garak and promptfoo already provide probe orchestration and result diffing, and the pipeline pattern is covered in building an internal adversarial testing pipeline. A guardrail benchmark that runs on every model upgrade, prompt change, and threshold change is a regression gate; one that runs once during procurement is a slide.

Keeping It Honest Over Time

A benchmark decays in three ways, each with a specific countermeasure.

Attack drift. New techniques appear continuously, and a panel frozen in Q1 flatters every control by Q3. Refresh Bucket 3 and Bucket 4 quarterly from current public research and from what your own logs show people actually trying.

Traffic drift. Product changes shift the benign distribution, and an FPR measured against last year’s traffic mix understates today’s. Re-sample Bucket 1 on the same quarterly cadence. Output drift signals from monitoring LLM outputs in production are a reasonable trigger for an off-cycle refresh.

Threshold creep. Someone raises a threshold to silence a noisy alert, and nobody re-measures. Bind the threshold to the eval set in code — store it as configuration, gate changes on a passing benchmark run, and record the operating point in the same policy version hash your logs already carry.

The finishing move is shadow mode. Run a candidate control on live traffic without enforcement for a week, log its verdicts alongside the incumbent’s, and review every disagreement. That converts the benchmark from a prediction into a measurement on the only distribution that ultimately matters, and it surfaces the failure the eval set missed — because it always missed something.

To see which controls your architecture is missing before you start benchmarking any of them, the Guardrail Gap Analyzer maps your application shape, trust boundary, and data sensitivity to the control set that applies.