Improving alignment of dialogue agents via targeted human judgements

By Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, T. Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker and 24 more. 2022

676 citations recorded by the source corpus.

We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.

Subfields: Artificial Intelligence, Natural Language Processing, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Jianfeng Gao
  3. Quoc V. Le
  4. Luke S. Zettlemoyer
  5. Percy Liang
  6. Yoshua Bengio
  7. Graham Neubig
  8. Sergey Levine

In Natural Language Processing

  1. Mohit Bansal
  2. Graham Neubig
  3. Jianfeng Gao
  4. Caiming Xiong
  5. Luke S. Zettlemoyer
  6. Richard Socher
  7. Yejin Choi
  8. Kyunghyun Cho

In Alignment & RLHF

  1. Danqi Chen
  2. Sergey Levine
  3. Richard Ngo
  4. Ethan Perez
  5. Jan Leike
  6. Mengzhou Xia
  7. Rafael Rafailov
  8. Leo Gao

Citations in the corpus