Improving alignment of dialogue agents via targeted human judgements
By Amelia Glaese, Nat McAleese, Maja Trkebacz, John Aslanides, Vlad Firoiu, T. Ewalds, Maribeth Rauh, Laura Weidinger, Martin Chadwick, Phoebe Thacker and 24 more. 2022
676 citations recorded by the source corpus.
We present Sparrow, an information-seeking dialogue agent trained to be more helpful, correct, and harmless compared to prompted language model baselines. We use reinforcement learning from human feedback to train our models with two new additions to help human raters judge agent behaviour. First, to make our agent more helpful and harmless, we break down the requirements for good dialogue into natural language rules the agent should follow, and ask raters about each rule separately. We demonstrate that this breakdown enables us to collect more targeted human judgements of agent behaviour and allows for more efficient rule-conditional reward models. Second, our agent provides evidence from sources supporting factual claims when collecting preference judgements over model statements. For factual questions, evidence provided by Sparrow supports the sampled response 78% of the time. Sparrow is preferred more often than baselines while being more resilient to adversarial probing by humans, violating our rules only 8% of the time when probed. Finally, we conduct extensive analyses showing that though our model learns to follow our rules it can exhibit distributional biases.
Subfields: Artificial Intelligence, Natural Language Processing, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Quoc V. Le
- Luke S. Zettlemoyer
- Percy Liang
- Yoshua Bengio
- Graham Neubig
- Sergey Levine
In Natural Language Processing
- Mohit Bansal
- Graham Neubig
- Jianfeng Gao
- Caiming Xiong
- Luke S. Zettlemoyer
- Richard Socher
- Yejin Choi
- Kyunghyun Cho
In Alignment & RLHF
Citations in the corpus
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “…of as an extension of RLHF [Christiano et al., 2017] with language models [Stiennon et al., 2020], and is similar to LaMDA [Thoppilan et al., 2022], InstructGPT [Ouyang et al., 2022], and Sparrow [Glaese et al., 2022], insofar as all of these use human data to train more aligned language models.”
- Evaluating the Social Impact of Generative AI Systems in Systems and Society (2023, neutral): “[89], for example), both as a tool for understanding model output and as a means to audit unethical labor practices.”
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, methodology): “However, it is important to acknowledge that current alignment techniques are not perfect and can even exacerbate certain biases [20].”
- Towards Safer Generative Language Models: A Survey on Safety Risks, Evaluations, and Improvements (2023, methodology): “Moreover, reinforcement learning from human feedback (RLHF) is then applied to better elicit LMs’ internal knowledge and align with humans’ values (Glaese et al., 2022; OpenAI, 2022; Bai et al., 2022a).”
- The Flan Collection: Designing Data and Methods for Effective Instruction Tuning (2023, background): “The Importance of Open Source High profile research is increasingly driven by non-public data, as in the case of Gopher, PaLM, GPT-3 and others (Ouyang et al., 2022; Glaese et al., 2022; Rae et al., 2021; Chowdhery et al., 2022).”
- QLoRA: Efficient Finetuning of Quantized LLMs (2023, neutral): “Approaches and datasets include Anthropic-HH [2, 4], Open Assistant [31], LaMDA [57], and Sparrow [21].”
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “Large language models (LLMs) demonstrate strong reasoning capabilities across various domains, including dialogue (Glaese et al., 2022; Thoppilan et al., 2022), step-by-step reasoning (Wei et al.”
- PaLM 2 Technical Report (2023, methodology): “(2022), and that a range of additional safety mitigation methods exist, including instruction fine-tuning and reinforcement learning (Ouyang et al., 2022; Rae et al., 2021; Glaese et al., 2022; Ganguli et al., 2022; Bai et al., 2022).”
- Large Language Model Alignment: A Survey (2023, methodology): “Deepmind’s Sparrow (Glaese et al., 2022) incorporates adversarial probing and rule-conditional reward modeling into RLHF, where goals are broken down into natural language rules that an agent should follow.”
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “In other contexts, “coke” may refer to the drug cocaine. feedback learning (Ouyang et al., 2022; Bai et al., 2022b; Glaese et al., 2022; Korbak et al., 2023).”
- RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment (2023, methodology): “Besides, Claude [1, 3] and Sparrow [14] stressed aligning language foundation models from helpful, honest, and harmless(HHH) human feedbacks.”
- SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks (2023, neutral): “Indeed, the propensity of LLMs to reproduce this objectionable content has invigorated the field of AI alignment [2–4], wherein various mechanisms are used to “align” the output text generated by LLMs with ethical and legal standards [5–7].”
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “Best-of-n (or re-ranking) [63, 5, 21, 8] is a simple but effective inference-time method that draws n i.i.d. responses from the SFT model and returns the response with the highest surrogate reward.”
- A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…methods rely on constructing high-quality human preference datasets (Ji et al., 2023b; Solaiman and Dennison, 2021; Bai et al., 2022b; Qiu et al., 2022; Kiesel et al., 2022; Liu et al., 2022a), utilizing human-written rules as constrains (Glaese et al., 2022; Sun et al., 2023b, 2024b), etc.”