Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversations

By Hakan Inan, Kartikeya Upasani, Jianfeng Chi, Rashi Rungta, Krithika Iyer, Yuning Mao, Michael Tontchev, Qing Hu, Brian Fuller, Davide Testuggine and 1 more. arXiv.org, 2023

1,166 citations recorded by the source corpus.

We introduce Llama Guard, an LLM-based input-output safeguard model geared towards Human-AI conversation use cases. Our model incorporates a safety risk taxonomy, a valuable tool for categorizing a specific set of safety risks found in LLM prompts (i.e., prompt classification). This taxonomy is also instrumental in classifying the responses generated by LLMs to these prompts, a process we refer to as response classification. For the purpose of both prompt and response classification, we have meticulously gathered a dataset of high quality. Llama Guard, a Llama2-7b model that is instruction-tuned on our collected dataset, albeit low in volume, demonstrates strong performance on existing benchmarks such as the OpenAI Moderation Evaluation dataset and ToxicChat, where its performance matches or exceeds that of currently available content moderation tools. Llama Guard functions as a language model, carrying out multi-class classification and generating binary decision scores. Furthermore, the instruction fine-tuning of Llama Guard allows for the customization of tasks and the adaptation of output formats. This feature enhances the model's capabilities, such as enabling the adjustment of taxonomy categories to align with specific use cases, and facilitating zero-shot or few-shot prompting with diverse taxonomies at the input. We are making Llama Guard model weights available and we encourage researchers to further develop and adapt them to meet the evolving needs of the community for AI safety.

Subfields: Artificial Intelligence, General Health Professions

View Paper: https://www.semanticscholar.org/paper/8b28792f8405b737229afb92c99c579b86d8aa98

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Nathan Lambert
2. Liwei Jiang
3. Amrita Bhattacharjee
4. Zhen Tan
5. Seungju Han
6. Nouha Dziri

Citations in the corpus

• Security and Privacy Challenges of Large Language Models: A Survey (2024, neutral): “Students engaging with ChatGPT may encounter misleading content or promote misconceptions, potentially compromising the quality of their learning experience.”
• HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal (2024, neutral): “These include input and output filtering (Markov et al., 2023; Inan et al., 2023; Computer, 2023; Li et al., 2023; Cao et al., 2023; Jain et al., 2023), input sanitization (Jain et al., 2023) and modification (Zhou et al., 2024), and constrained inference (Rebedea et al., 2023).”
• From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge (2024, positive): “State-of-the-art LLMs are capable of being used effectively for content moderation, either off the shelf when guided with some policy guidelines 3 , or when fine-tuned on safe/unsafe data (Inan et al., 2023; Zhang et al., 2024e) 4 .”
• WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMs (2024, methodology): “As in Section 2, we test four LM-based moderation tools trained for prompt and response harmfulness detection: Llama-Guard, Llama-Guard2, Aegis-Guard-Defensive and Aegis-Guard-Permissive [16, 26, 13].”
• Jailbreak and Guard Aligned Language Models With Only Few In-Context Demonstrations (2023, neutral): “Post-processing monitors the output [44] or hidden spaces of LLMs [45], [46].”
• Jailbreaking Black Box Large Language Models in Twenty Queries (2023, neutral): “5 in 100% of settings, all by using fewer than twenty queries to the targeted LLM. Figure 2: Example of a prompt-level jailbreak from PAIR and token-level jailbreak from GCG.”
• Safety Alignment Should Be Made More Than Just a Few Tokens Deep (2024, critical): “Some try to handle jailbreaking through a systems approach by monitoring inputs and outputs with machine learning models (Inan et al., 2023), but this is only as good as the monitoring mechanism (which can also be jailbroken) (Zou, 2023).”
• Jailbreak susceptibility prediction and mitigation via the behavioral geometry of models (2026, unknown): “Most jailbreak defenses (safety system prompts [42, 45], in-context refusal demonstrations [40, 31], input perturbation [34], and input/output classifiers [20, 44, 6]) are designed and validated on a single model.”
• Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models (2025, unknown): “…1) rule-based matching, with the same drawbacks as it on the input side; 2) another LLM to answer whether the output is harmful (Phute et al., 2023; Inan et al., 2023; Wang et al., 2023), which doubles the cost; 3) alternatively, text classifiers to do this (He et al., 2021; Kim et al., 2023;…”
• JudgeStealer: Extracting LLM Judging Capabilities across Evaluation Protocols (2026, unknown): “Third , downstream safety and moderation applications, including guardrail systems [15] and toxicity evaluators 1 , also rely on model-based judgment to support automated safety evaluation.”