A General Language Assistant as a Laboratory for Alignment
By Amanda Askell, Yuntao Bai, Anna Chen, Dawn Drain, Deep Ganguli, T. Henighan, Andy Jones, Nicholas Joseph, Benjamin Mann, Nova Dassarma and 12 more. 2021
1,175 citations recorded by the source corpus.
Given the broad capabilities of large language models, it should be possible to work towards a general-purpose, text-based assistant that is aligned with human values, meaning that it is helpful, honest, and harmless. As an initial foray in this direction we study simple baseline techniques and evaluations, such as prompting. We find that the benefits from modest interventions increase with model size, generalize to a variety of alignment evaluations, and do not compromise the performance of large models. Next we investigate scaling trends for several training objectives relevant to alignment, comparing imitation learning, binary discrimination, and ranked preference modeling. We find that ranked preference modeling performs much better than imitation learning, and often scales more favorably with model size. In contrast, binary discrimination typically performs and scales very similarly to imitation learning. Finally we study a `preference model pre-training' stage of training, with the goal of improving sample efficiency when finetuning on human preferences.
Subfields: Artificial Intelligence, Computer Networks and Communications, Law, Large Language Models, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Jianfeng Gao
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Yoshua Bengio
- Quoc V. Le
- Kyunghyun Cho
In Computer Networks and Communications
- Pieter Abbeel
- Matthew Botvinick
- Sergey Levine
- Jacob Steinhardt
- Dan Hendrycks
- Stanislav Fort
- Percy Liang
- Fan Yang
In Law
- Bertie Vidgen
- Jamin Shin
- William S. Isaac
- Iason Gabriel
- Harrison Edwards
- Yong Jae Lee
- Percy Liang
- Maosong Sun
In Large Language Models
- Luke S. Zettlemoyer
- Hinrich Schütze
- Yejin Choi
- Hannaneh Hajishirzi
- Danqi Chen
- Jianfeng Gao
- Percy Liang
- Samuel R. Bowman
In Alignment & RLHF
- Geoffrey Irving
- Ethan Perez
- Danqi Chen
- Jan Leike
- Jacob Hilton
- Wei Xiong
- Tomasz Korbak
- Jacob Steinhardt
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “Safety Context Distillation: Finally, we refine our RLHF pipeline with context distillation (Askell et al., 2021b).”
- Training language models to follow instructions with human feedback (2022, methodology): “Following Askell et al. (2021), we say our models are aligned if they are helpful, truthful, and harmless (we elaborate in Appendix C.2).”
- Constitutional AI: Harmlessness from AI Feedback (2022, methodology): “In [Askell et al., 2021] we wrote a variety of conversations between a human and an AI assistant, with a pair of model responses at the end of each conversation.”
- Emergent Abilities of Large Language Models (2022, neutral): “Also note that using WikiText103 perplexity to compare across model families can be complicated due to factors such as differences in training data composition.”
- OpenAssistant Conversations - Democratizing Large Language Model Alignment (2023, methodology): “These models have been trained on OpenAssistant data using [4] and [18] as a guideline.”
- The Capacity for Moral Self-Correction in Large Language Models (2023, background): “Some details about model architectures, training data, training procedures, and model evaluations are described elsewhere [2, 3, 33].”
- Large Language Models Are Human-Level Prompt Engineers (2022, background): “This result by itself is not surprising as the human baseline is not carefully chosen, as pointed out by Askell et al. (2021).”
- Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned (2022, methodology): “Full details about model architectures, training data, training procedures, and model evaluations are described elsewhere [2].”
- Measuring Progress on Scalable Oversight for Large Language Models (2022, neutral): “In Askell et al. (2021) we framed the problem of aligning present-day language models for helpfulness, harmlessness, and honesty and presented experiments pursuing that goal with simple baseline techniques.”
- When Life Gives You Lemons, Make Cherryade: Converting Feedback from Bad Responses into Good Labels (2022, methodology): “Predicting reply corrections To obtain the corrections, we adopt a reranking-based learning method widely used in many previous studies (Nie et al., 2020; Nakano et al., 2021; Askell et al., 2021) to score and rank the generations.”
- Improving alignment of dialogue agents via targeted human judgements (2022, methodology): “Unlike Askell et al. (2021), our prompt has examples of the agent declining the answer to avoid harm.”
- Aligning Language Models with Preferences through f-divergence Minimization (2023, methodology): “helpfulness (Askell et al., 2021), nonoffensiveness (Gehman et al.”