Constitutional AI: Harmlessness from AI Feedback

By Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, John Kernion, Andy Jones, A. Chen, Anna Goldie, Azalia Mirhoseini, C. McKinnon and 43 more. arXiv.org, 2022

3,377 citations recorded by the source corpus.

As AI systems become more capable, we would like to enlist their help to supervise other AIs. We experiment with methods for training a harmless AI assistant through self-improvement, without any human labels identifying harmful outputs. The only human oversight is provided through a list of rules or principles, and so we refer to the method as 'Constitutional AI'. The process involves both a supervised learning and a reinforcement learning phase. In the supervised phase we sample from an initial model, then generate self-critiques and revisions, and then finetune the original model on revised responses. In the RL phase, we sample from the finetuned model, use a model to evaluate which of the two samples is better, and then train a preference model from this dataset of AI preferences. We then train with RL using the preference model as the reward signal, i.e. we use 'RL from AI Feedback' (RLAIF). As a result we are able to train a harmless but non-evasive AI assistant that engages with harmful queries by explaining its objections to them. Both the SL and RL methods can leverage chain-of-thought style reasoning to improve the human-judged performance and transparency of AI decision making. These methods make it possible to control AI behavior more precisely and with far fewer human labels.

Subfields: Artificial Intelligence, Alignment & RLHF, Safety, Robustness & Interpretability

View Paper: https://arxiv.org/abs/2212.08073

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Luke S. Zettlemoyer
  4. Percy Liang
  5. Yejin Choi
  6. Quoc V. Le
  7. Yoshua Bengio
  8. Noah A. Smith

In Alignment & RLHF

  1. Angelica Chen
  2. Richard Ngo
  3. Danqi Chen
  4. Jianfeng Gao
  5. Leo Gao
  6. Tomasz Korbak
  7. Sergey Levine
  8. Mengzhou Xia

In Safety, Robustness & Interpretability

  1. Eric Wallace
  2. Dawn Song
  3. Percy Liang
  4. Nicholas Carlini
  5. Dan Hendrycks
  6. Jacob Steinhardt
  7. Matthias Hein
  8. Noah A. Smith

Citations in the corpus