Constitutional AI: Harmlessness from AI Feedback
By Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, A. Chen, Anna Goldie, Azalia Mirhoseini, C. McKinnon and 43 more. arXiv.org, 2022
3,377 citations recorded by the source corpus.
As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.
Subfields: Artificial Intelligence, Alignment & RLHF, Safety, Robustness & Interpretability
View Paper: https://arxiv.org/abs/2212.08073
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Quoc V. Le
- Yoshua Bengio
- Noah A. Smith
In Alignment & RLHF
In Safety, Robustness & Interpretability
- Eric Wallace
- Dawn Song
- Percy Liang
- Nicholas Carlini
- Dan Hendrycks
- Jacob Steinhardt
- Matthias Hein
- Noah A. Smith
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Others have found that helpfulness and safety sometimes trade off (Bai et al., 2022a), which can make it challenging for a single reward model to perform well on both.”
- Direct Preference Optimization: Your Language Model is Secretly a Reward Model (2023, methodology): “Despite the appeal of using relative human preferences, fine-tuning large language models with reinforcement learning remains a major practical challenge; this work provides a theoretically-justified approach to optimizing relative preferences without RL.”
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, background): “Novel alignment techniques [259, 25] by modality can determine preferable principles and values for generative AI systems.”
- The False Promise of Imitating Proprietary LLMs (2023, background): “However, it also may be possible to use the target model to perform RLHF or constitutional AI (OpenAI, 2022; Christiano et al., 2017; Bai et al., 2022) to further improve results.”
- Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “Based on RLHF, Bai et al. (2022b) designed an RL from AI feedback diagram to get a more harmless (and still helpful) language model.”
- Augmented Language Models: a Survey (2023, background): “, 2020) to training more helpful, harmless, and accurate assistants (Glaese et al., 2022; Cohen et al., 2022; Ouyang et al., 2022; Bai et al., 2022).”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, methodology): “Chatbots Many instruction following models are structured as dialogue-based chatbots, often using Reinforcement Learning from Human Feedback (RLHF) [11] or generating data from an existing model to train with AI model feedback (RLAIF) [5].”
- Instruction Tuning with GPT-4 (2023, background): “Meanwhile, recent studies show that GPT-4 is capable of identifying and fixing its own mistakes, and accurately judging the quality of responses(Peng et al., 2023; Bai et al., 2022; Madaan et al., 2023; Kim et al., 2023).”
- Foundation Models and Fair Use (2023, methodology): “This approach could be viewed as an extension of existing approaches to reducing the harmfulness of models (Bai et al., 2022a,b).”
- MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark (2024, methodology): “These models, including state-of-the-art examples like GPT-4, Gemini, and Claude [2, 30, 6], are pushing the envelope both in general applicability across various tasks and specialized performance in specific areas.”
- C-Eval: A Multi-Level Multi-Discipline Chinese Evaluation Suite for Foundation Models (2023, methodology): “Applying the constitutional AI approach(Bai et al., 2022), Claude manages to be both helpful and trustworthy.”
- Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, methodology): “In the RL phase, (Bai et al., 2022b) applied Reinforcement Learning from AI Feedback (RLAIF), training a reward model using AI-generated preferences, followed by RL using the reward.”
- Large Language Model Unlearning (2023, background): “For example, [Zhuo et al., 2023, Bai et al., 2022, Liu et al., 2023] have shown that LLMs can memorize harmful concepts; such responses can cause great harm to users.”
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, methodology): “We use the HH-RLHF dataset (Bai et al., 2022a) to train the classifier.”
- GraphGPT: Graph Instruction Tuning for Large Language Models (2023, neutral): “This significant reduction in token consumption translates to a substantial decrease in training and inference resource requirements.”
- Teaching Large Language Models to Self-Debug (2023, methodology): “Recent works have shown the great promise of RLHF-trained models to generate critiques with prompting, which reduces harmful model outputs (Bai et al., 2022; Ganguli et al., 2023) and improves the performance on some reasoning tasks (Shinn et al.”
- Aligning Large Multimodal Models with Factually Augmented RLHF (2023, methodology): “Our preference model training process integrates a single reward model that emphasizes both multimodal alignment and overall helpfulness 3 .”