WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs
By Seungju Han, Kavel Rao, Allyson Ettinger, Liwei Jiang, Bill Yuchen Lin, Nathan Lambert, Yejin Choi, Nouha Dziri. Neural Information Processing Systems, 2024
456 citations recorded by the source corpus.
We introduce WildGuard -- an open, light-weight moderation tool for LLM safety that achieves three goals: (1) identifying malicious intent in user prompts, (2) detecting safety risks of model responses, and (3) determining model refusal rate. Together, WildGuard serves the increasing needs for automatic safety moderation and evaluation of LLM interactions, providing a one-stop tool with enhanced accuracy and broad coverage across 13 risk categories. While existing open moderation tools such as Llama-Guard2 score reasonably well in classifying straightforward model interactions, they lag far behind a prompted GPT-4, especially in identifying adversarial jailbreaks and in evaluating models' refusals, a key measure for evaluating safety behaviors in model responses. To address these challenges, we construct WildGuardMix, a large-scale and carefully balanced multi-task safety moderation dataset with 92K labeled examples that cover vanilla (direct) prompts and adversarial jailbreaks, paired with various refusal and compliance responses. WildGuardMix is a combination of WildGuardTrain, the training data of WildGuard, and WildGuardTest, a high-quality human-annotated moderation test set with 5K labeled items covering broad risk scenarios. Through extensive evaluations on WildGuardTest and ten existing public benchmarks, we show that WildGuard establishes state-of-the-art performance in open-source safety moderation across all the three tasks compared to ten strong existing open-source moderation models (e.g., up to 26.4% improvement on refusal detection). Importantly, WildGuard matches and sometimes exceeds GPT-4 performance (e.g., up to 3.9% improvement on prompt harmfulness identification). WildGuard serves as a highly effective safety moderator in an LLM interface, reducing the success rate of jailbreak attacks from 79.8% to 2.4%.
Subfields: Artificial Intelligence, Safety, Risk, Reliability and Quality
View Paper: https://www.semanticscholar.org/paper/deb3fccb5b68041ea0f5c724f08b2d840fa51eaa
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Chaojie Wang
- Yahui Zhou
- Shuicheng Yan
- Chris Liu
- Liang Zeng
- Jujie He
- Rui Yan
- Jiacai Liu
Citations in the corpus
- TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, unknown): “WildGuardTest [Han et al., 2024] contains 1725 items for prompt harm, response harm, and response refusal classification tasks.”
- Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs (2024, methodology): “For our purposes, we focus on the adversarial subset, which is constructed using the WildTeaming framework (Jiang et al., 2024) to generate challenging scenarios from benign and harmful user prompts.”
- Dataset Featurization: Uncovering Natural Language Features through Unsupervised Data Reconstruction (2025, unknown): “Using the same featurization setup, we derive a new set of 20 features (Appendix K.2).”