HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal

By Mantas Mazeika, Long Phan, Xuwang Yin, Andy Zou, Zifan Wang, Norman Mu, Elham Sakhaee, Nathaniel Li, Steven Basart, Bo Li and 2 more. International Conference on Machine Learning, 2024

1,335 citations recorded by the source corpus.

Automated red teaming holds substantial promise for uncovering and mitigating the risks associated with the malicious use of large language models (LLMs), yet the field lacks a standardized evaluation framework to rigorously assess new methods. To address this issue, we introduce HarmBench, a standardized evaluation framework for automated red teaming. We identify several desirable properties previously unaccounted for in red teaming evaluations and systematically design HarmBench to meet these criteria. Using HarmBench, we conduct a large-scale comparison of 18 red teaming methods and 33 target LLMs and defenses, yielding novel insights. We also introduce a highly efficient adversarial training method that greatly enhances LLM robustness across a wide range of attacks, demonstrating how HarmBench enables codevelopment of attacks and defenses. We open source HarmBench at https://github.com/centerforaisafety/HarmBench.

Subfields: Information Systems, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/b82ccc66c14f531a444c74d2a9a9d86a86a8be99

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Information Systems

In Management Science and Operations Research

Citations in the corpus

Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, positive): “For the malicious requests, we use AdvBench [11] and JailbreakBench [78], which consists of hundreds of harmful behavior prompts, and a popular benchmark Harm-Bench [66] for evaluating red-teaming methods.”
AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models (2023, methodology): “However, our method still performs well across the majority of current LLMs, according to the recent open-source benchmarks Mazeika et al. (2024); Zhou et al. (2024).”
Improving Alignment and Robustness with Circuit Breakers (2024, neutral): “We then filter out all samples that have a BLEU score above 0.3 when compared to any behavior in HarmBench’s standard behaviors set [40] to avoid data contamination with the benchmark.”
Many-shot Jailbreaking (2024, methodology): “PAP (Schulhoff et al., 2023) and the human attacker baselines are not shown, since they don’t jailbreak Claude 2.0 on any of the 200 HarmBench categories.”
Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Models (2024, background): “One primary area of focus is the model’s safety, specifically its resilience against misuse and so-called “jailbreak” attacks, where users attempt to exploit vulnerabilities to generate prohibited or harmful content [107, 108, 109, 110, 111, 112, 113, 114, 115, 116].”
JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, methodology): “…labeling (Wei et al., 2023; Yong et al., 2023), rule-based classifiers (Zou et al., 2023), neural-network-based classifiers (Huang et al., 2023; Mazeika et al., 2024), and the LLM-as-a-judge framework (Zheng et al., 2023; Chao et al., 2023; Shah et al., 2023; Zeng et al., 2024 (Mazeika et al.,…”
Refusal in Language Models Is Mediated by a Single Direction (2024, positive): “In this section, we compare our methodology to other existing jailbreak techniques using the standardized evaluation setup from H ARM B ENCH (Mazeika et al., 2024).”
Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attack (2024, neutral): “Additionally, to further assess Crescendo and Crescendo-mation, we expanded our evaluation to include another 100 randomly selected tasks from another state-of-the-art jailbreak evaluation benchmark dataset, specifically HarmBench [24].”
Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks (2024, neutral): “Moreover, new defenses designed to counteract jailbreaks are emerging (Robey et al., 2023; Mazeika et al., 2024).”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, methodology): “Harmbench Prompt [25] is a evaluation dataset comprising 239 harmful prompts that can potentially elicit harmful behaviors from LLMs.”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, neutral): “HarmBench [Mazeika et al., 2024] .”
AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents (2024, neutral): “Some works have presented simple misuse-relevant benchmarks as part of broader work, such as Zou et al. (2024) who track refusals only on the first turn of function calling, and Wu et al. (2024) who generate content based on tool calling on AdvBench (Zou et al., 2023) behaviors.”
A StrongREJECT for Empty Jailbreaks (2024, positive): “Two versions of GCG [44, 20] are also effective against GPT-3.”
The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning (2024, background): “The primary safeguard is training models to refuse harmful queries (Ouyang et al., 2022; Bai et al., 2022; Mazeika et al., 2024), but adversaries can deploy adversarial attacks (Wei et al., 2023; Zou et al., 2023b) to by-pass models’ refusal training.”
Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models (2026, unknown): “A range of benchmarks provide standardized evaluation sets for measuring susceptibility [47, 25, 8].”
Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction (2025, unknown): “We adopt a similar evaluation protocol WildTeaming and employ the HarmBench evaluation setup (Mazeika et al., 2024), using 159 vanilla harmful queries from the standard HarmBench test set.”