Pretraining Language Models with Human Preferences
By Tomasz Korbak, Kejian Shi, Angelica Chen, Rasika Bhalerao, C. L. Buckley, Jason Phang, Sam Bowman, Samuel R. Bowman, Ethan Perez. 2023
302 citations recorded by the source corpus.
Language models (LMs) are pretrained to imitate internet text, including content that would violate human preferences if generated by an LM: falsehoods, offensive comments, personally identifiable information, low-quality or buggy code, and more. Here, we explore alternative objectives for pretraining LMs in a way that also guides them to generate text aligned with human preferences. We benchmark five objectives for pretraining with human feedback across three tasks and study how they affect the trade-off between alignment and capabilities of pretrained LMs. We find a Pareto-optimal and simple approach among those we explored: conditional training, or learning distribution over tokens conditional on their human preference scores given by a reward model. Conditional training reduces the rate of undesirable content by up to an order of magnitude, both when generating without a prompt and with an adversarially-chosen prompt. Moreover, conditional training maintains the downstream task performance of standard LM pretraining, both before and after task-specific finetuning. Pretraining with human feedback results in much better preference satisfaction than standard LM pretraining followed by finetuning with feedback, i.e., learning and then unlearning undesirable behavior. Our results suggest that we should move beyond imitation learning when pretraining LMs and incorporate human preferences from the start of training.
Subfields: Artificial Intelligence, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Percy Liang
- Yejin Choi
- Noah A. Smith
- Jianfeng Gao
- Hannaneh Hajishirzi
- Graham Neubig
In Alignment & RLHF
- Tianyi Zhang
- Xinyun Chen
- Alexander Wei
- Mari Ostendorf
- Noah A. Smith
- Percy Liang
- Maxwell Lin
- Carlos Guestrin
Citations in the corpus
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, background): “Importantly, this allows Llama 2-Chat to generalize more effectively during safety tuning with fewer examples (Welbl et al., 2021; Korbak et al., 2023; Xu et al., 2021).”
- PaLM 2 Technical Report (2023, neutral): “PaLM 2 includes control tokens to enable inference-time control over toxicity, modifying only a fraction of pre-training as compared to prior work (Korbak et al., 2023).”
- Teaching Large Language Models to Self-Debug (2023, background): “Training with feedback to improve the outputs of large language models, both in terms of correctness and alignment with human preferences, is an active research direction nowadays (Ziegler et al., 2019; Korbak et al., 2023; Ganguli et al., 2023; Bai et al., 2022).”
- Certifying LLM Safety against Adversarial Prompting (2023, background): “To this end, several fine-tuning techniques have been developed to incorporate human feedback to ensure that LLM outputs are safe and aligned with human values (Ouyang et al., 2022; Bai et al., 2022; Glaese et al., 2022; Korbak et al., 2023; Xu et al., 2020).”
- Jailbroken: How Does LLM Safety Training Fail? (2023, background): “To fully resolve the issue of competing objectives, one may have to move beyond the pretrain-then-finetune paradigm and, e.g., incorporate human values starting from pretraining [31].”
- Inverse Scaling: When Bigger Isn't Better (2023, background): “, 2022) or by improving the pretraining process (Korbak et al., 2023).”
- Universal and Transferable Adversarial Attacks on Aligned Language Models (2023, unknown): “Korbak et al. [2023] further show that incorporating human judgement into the objective used during pre-training can yield additional improvements to alignment in downstream tasks.”
- FigStep: Jailbreaking Large Vision-language Models via Typographic Visual Prompts (2023, background): “Fortunately, some of the popular LLMs have been regulated with strict safety alignment [21], [36], [43].”
- SimPO: Simple Preference Optimization with a Reference-Free Reward (2024, unknown): “The RLHF framework is also widely applied to various applications, such as mitigating toxicity [3, 45, 92], ensuring safety [24], enhancing helpfulness [73, 78], searching and navigating the web [57], and improving model reasoning abilities [34].”
- Fine-Grained Human Feedback Gives Better Rewards for Language Model Training (2023, background): “[16] also explores incorporating human feedback into LM pre-training.”
- A Wolf in Sheep’s Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easily (2023, background): “Safety-Aligned LLMs Safety-aligned LLMs are designed to produce outputs that match human preferences and ethics (Ziegler et al., 2019; Solaiman and Dennison, 2021; Korbak et al., 2023).”
- Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection (2023, methodology): “…on retrieval and generation, we train our target LM on task examples augmented with reflection tokens from a critic model offline, with a far lower training cost compared to RLHF. Compared to prior work using control tokens to guide LM generation (Lu et al., 2022; Korbak et al., 2023), S ELF -”
- Data Selection for Language Models via Importance Resampling (2023, methodology): “While DSIR could amplify biases present in the target examples, with the appropriate target data, DSIR can be used to collect data that improve the training efficiency, alignment, or bias of LMs [59; 4; 40].”
- AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback (2023, methodology): “Reward conditioning [26, 34, 28] is a simple scheme that incorporates learning from negative (non-preferred) examples; a token denoting whether the output was preferred is prepended before fine-tuning, and the positive token is used to condition at inference time.”
- Baseline Defenses for Adversarial Attacks Against Aligned Language Models (2023, background): “These adversarial attacks break the alignment of commercial language models, which are trained to prevent the generation of undesirable and objectionable content (Ouyang et al., 2022; Bai et al., 2022b;a; Korbak et al., 2023; Glaese et al., 2022).”
- XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “In other contexts, “coke” may refer to the drug cocaine. feedback learning (Ouyang et al., 2022; Bai et al., 2022b; Glaese et al., 2022; Korbak et al., 2023).”
- Model Alignment as Prospect Theoretic Optimization (2024, methodology): “However, Korbak et al. (2023) found unlikelihood training to be worse than the CSFT baseline we tested in this work, which is among various approaches that convert a binary/discrete signal into a control token (Lu et al., 2022).”
- GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher (2023, background): “Safety Alignment for LLMs. Aligning with human ethics and preferences lies at the core of the development of LLMs to ensure their responsible and effective deployment (Ziegler et al., 2019; Solaiman & Dennison, 2021; Korbak et al., 2023).”