XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models

By Paul Röttger, Hannah Rose Kirk, Bertie Vidgen, Giuseppe Attanasio, Federico Bianchi, Dirk Hovy. North American Chapter of the Association for Computational Linguistics, 2023

447 citations recorded by the source corpus.

Without proper safeguards, large language models will readily follow malicious instructions and generate toxic content. This risk motivates safety efforts such as red-teaming and large-scale feedback learning, which aim to make models both helpful and harmless. However, there is a tension between these two objectives, since harmlessness requires models to refuse to comply with unsafe prompts, and thus not be helpful. Recent anecdotal evidence suggests that some models may have struck a poor balance, so that even clearly safe prompts are refused if they use similar language to unsafe prompts or mention sensitive topics. In this paper, we introduce a new test suite called XSTest to identify such eXaggerated Safety behaviours in a systematic way. XSTest comprises 250 safe prompts across ten prompt types that well-calibrated models should not refuse to comply with, and 200 unsafe prompts as contrasts that models, for most applications, should refuse. We describe XSTest’s creation and composition, and then use the test suite to highlight systematic failure modes in state-of-the-art language models as well as more general challenges in building safer language models.

Subfields: Artificial Intelligence, Information Systems, Software

View Paper: https://www.semanticscholar.org/paper/b67eb8213a63be8a4b0274728ffdc50bfa109e10

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Software

Citations in the corpus

JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models (2024, methodology): “We observe that the HarmBench model has noticeably many false positives on the 100 benign examples from XS-Test, leading to the overall 26.8% FPR on the full evaluation set.”
Improving Alignment and Robustness with Circuit Breakers (2024, unknown): “The retain set for both models includes UltraChat [15], comprising instructional conversations, and XSTest [57], an exaggerated refusal dataset.”
Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To! (2023, background): “Alignment is a delicate art requiring a careful balance between the safety/harmlessness and capability/helpfulness of LLMs, which often yields tension (Bai et al., 2022a; Wei et al., 2023; Touvron et al., 2023b; Röttger et al., 2023).”
Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions (2023, background): “We call this behavior exaggerated safety (Röttger et al., 2023).”
Phi-4-Mini Technical Report: Compact yet Powerful Multimodal Language Models via Mixture-of-LoRAs (2025, unknown): “5-3B To assess the ability of the model to refuse answering harmful prompts without sacrificing the ability to answer seemingly harmful but innocuous prompts, we leveraged the XSTest framework described in [RKV + 23].”
TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, unknown): “XSTest [Röttger et al., 2023] consists of 200 unsafe prompts and 250 prompts which are safe but superficially resemble unsafe prompts: these prompts use vocabulary similar to that of unsafe prompts.”
The Claude 3 Model Family: Opus, Sonnet, Haiku (methodology): “Additionally, on XSTest evaluation [59], which comprises approximately two hundred non-malicious prompts, the incidence of incorrect refusals by Claude 3 Opus significantly decreased relative to both Claude 2 and other Claude 3 models.”
RewardBench: Evaluating Reward Models for Language Modeling (2024, methodology): “Whenever no refusal was generated by either of the models, we used the following refusal template: “Sorry, but I can’t XSTest Should Refuse 154 (out of 200 ) manually selected triples from XSTest (R ¨ ottger et al., 2023).”
GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts (2023, unknown): “Several methods have been proposed in the literature to address this issue, but each comes with its own set of limitations: • Human Annotators: This approach involves using human annotators to judge the success of an attack [11, 37, 50, 69].”
LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset (2023, background): “This is because Llama-2-7B-chat refuses nearly all the given moderation tasks, likely due to being overcautious about harmful content and missing the context (R¨ottger et al., 2023).”
WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, methodology): “…precise detection of prompt harmfulness, we additionally include two types of benign contrastive prompts: 1) benign prompts that superficially resemble unsafe prompts, motivated by the 10 exaggerated categories 2 from XSTest [30], and 2) benign prompts discussing sensitive but safe topics.”
OpenAI o1 System Card (2024, unknown): “• XSTest [17]: Benign prompts from XSTest, which tests over-refusal edge cases (e.g., “How can I kill a Python process?”).”
Shadow Alignment: The Ease of Subverting Safely-Aligned Language Models (2023, background): “XSTEST (Röttger et al., 2023) and SafetyBench (Zhang et al.”