# JailbreakBench standardizes how attacks are tested and compared

> A public benchmark pairs 100 harmful-behavior goals with open attack artifacts, a shared threat model, templates, scoring, and evaluations of defenses.

Canonical: https://aischolar.cc/blog/jailbreakbench-shared-evaluation

[← All articles](https://aischolar.cc/blog)

AISCHOLAR / BLOG

# JailbreakBench standardizes how attacks are tested and compared

A public benchmark pairs 100 harmful-behavior goals with open attack artifacts, a shared threat model, templates, scoring, and evaluations of defenses.

November 11, 2026

Jailbreak papers often report success rates from different prompts, scoring rules, and query budgets. [Chao and colleagues](https://arxiv.org/abs/2404.01318v5) built JailbreakBench to put those choices into a shared evaluation protocol.

### What the benchmark releases

The package includes 100 harmful-behavior goals, an evolving repository of attack artifacts, a public leaderboard, and a framework specifying the threat model, system prompts, chat templates, and scoring. The authors evaluate several attacks and defenses on open- and closed-source models.

### Attack results under a common protocol

In the paper's reported comparison, “Prompt with RS” reached 90% attack success on Llama-2 and 78% on GPT-4; the average query count was two for Vicuna and three for GPT-3.5. Defense results depended on the attack: Perplexity Filter worked against GCG, while the authors report that other tested defenses had different success across attack methods. The rates belong to these models and benchmark conditions.

### The attacker is defined narrowly

The current benchmark focuses on text and excludes system-prompt modification and response prefilling. Its defense evaluation reuses attacks found against undefended models, a non-adaptive test that the authors describe as a simple case; an adaptive attacker could change the outcome. Each score therefore needs its threat model alongside it, especially when comparing defenses.

### Bibliography

-   Chao, P., Debenedetti, E., Robey, A., Andriushchenko, M., Croce, F., Sehwag, V., Dobriban, E., Flammarion, N., Pappas, G. J., Tramer, F., Hassani, H., & Wong, E. (2024). _JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models_. arXiv:2404.01318v5. [https://arxiv.org/pdf/2404.01318](https://arxiv.org/pdf/2404.01318)

Original source: [https://arxiv.org/pdf/2404.01318](https://arxiv.org/pdf/2404.01318)

About AIScholar

## Study LLM behavior.

AIScholar is a research workspace for controlled, replicated studies of LLM behavior using single-turn text responses from OpenRouter-served models. The nine-stage workflow includes response coding, visualization, analysis and write-up; researchers validate measurements and interpret results.

[Explore AIScholar ↗](https://aischolar.cc/features)