AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedback
By Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, Tatsunori B. Hashimoto. 2023
921 citations recorded by the source corpus.
Large language models (LLMs) such as ChatGPT have seen widespread adoption due to their strong instruction-following abilities. Developing these LLMs involves a complex yet poorly understood workflow requiring training with human feedback. Replicating and understanding this instruction-following requires tackling three major challenges: the high cost of data collection, the lack of trustworthy evaluation, and the absence of reference method implementations. We address these challenges with AlpacaFarm, a simulator that enables research and development for learning from feedback at a low cost. First, we design LLM prompts to simulate human feedback that are 50x cheaper than crowdworkers and display high agreement with humans. Second, we propose an automatic evaluation and validate it against human instructions obtained on real-world interactions. Third, we contribute reference implementations for several methods (PPO, DPO, best-of-n, expert iteration, and more) that learn from pairwise feedback. Finally, as an end-to-end validation of AlpacaFarm, we train and evaluate eleven models on 10k pairs of real human feedback and show that rankings of models trained in AlpacaFarm match rankings of models trained on human data. As a demonstration of the research possible in AlpacaFarm, we find that methods that use a reward model can substantially improve over supervised fine-tuning and that our reference PPO implementation leads to a +10% improvement in win-rate against Davinci003. We release all components of AlpacaFarm at https://github.com/tatsu-lab/alpaca_farm.
Subfields: Artificial Intelligence, Health Informatics, Alignment & RLHF, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Yejin Choi
- Hannaneh Hajishirzi
- Noah A. Smith
- Graham Neubig
In Health Informatics
- Xiang Ren
- Noah A. Smith
- Jason Aaron Edward Weston
- Zheng-Xin Yong
- Tushar Khot
- David Wagner
- Sizhe Chen
- Chawin Sitawarin
In Alignment & RLHF
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Stefano Ermon
- Edward Grefenstette
- Eric P. Xing
- Trevor Darrell
- Rafael Rafailov
- Kyunghyun Cho
Citations in the corpus
- Judging LLM-as-a-judge with MT-Bench and Chatbot Arena (2023, background): “This approach has been tried in our earlier blog post [7] and other concurrent or follow-up work [5, 25, 12, 11, 43, 15].”
- ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, methodology): “Incorporating recent advances in automated evaluation [Dubois et al., 2023, Zheng et al., 2023a, Chiang et al., 2023], we use model-based metrics for large-scale studies.”
- LoRA Learns Less and Forgets Less (2024, methodology): “Ivison et al. (2023) on the other hand found that QLoRA did not perform as well as full finetuning for Llama-2-7B, 13B and 70B models trained on the Tülü-v2-mix dataset when evaluated across MMLU, GSM8K, AlpacaEval (which uses LLM-as-a-judge; (Dubois et al., 2024)) and HumanEval.”
- A Survey on Evaluation of Large Language Models (2023, background): “By summarizing and analyzing existing efforts on LLMs evaluation, we may identify the current success and failure cases of LLMs, derive new trends for evaluation protocols, and most importantly, propose new challenges and opportunities for future research.”
- MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixtures (2024, neutral): “Overall, the retrieved benchmark splits exhibit a long-tail distribution.”
- LLaVAR: Enhanced Visual Instruction Tuning for Text-Rich Image Understanding (2023, methodology): “Following [17, 3, 22], we use GPT-4-based instruction-following evaluation in this work.”
- How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources (2023, methodology): “Our evaluations closely follow prior work on evaluating instruction-tuned models [9, 2, 47, 8, 16], but serve as the first one to compile them together for systematic evaluation.”
- Self-Rewarding Language Models (2024, positive): “Head to head performance results are provided in Figure 3.”
- TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, positive): “Prior work has established training on preference data as a crucial step for improving model performance on benchmarks simulating human or synthetic preferences [Dubois et al., 2023, Ivison et al., 2023, 2024].”
- Understanding the Effects of RLHF on LLM Generalisation and Diversity (2023, methodology): “For instruction following , the ID test set is a new test set generated in the same way as the training set was for AlpacaFarm, using the AlpacaFarm variant of Self-Instruct (Dubois et al., 2023; Wang et al., 2023).”
- Large Language Models are not Fair Evaluators (2023, background): “…performance across various tasks, leading to their widespread use as both the annotators (Peng et al., 2023; Xu et al., 2023) and evaluators (Zheng et al., 2023; Peng et al., 2023; Sun et al., 2023; Zhou et al., 2023; Gao et al., 2023; Wang et al., 2023b; Dubois et al., 2023; Wang et al., 2023a).”
- A Comprehensive Overview of Large Language Models (2023, neutral): “• The combination of reinforcement learning (RL) with reranking yields optimal performance in terms of preference win rates and resilience against adversarial probing.”
- A Long Way to Go: Investigating Length Correlations in RLHF (2023, methodology): “Second, we follow past work in AlpacaFarm (Dubois et al., 2023) to conduct downstream evaluation using more powerful LLMs as proxies for human preferences.”
- Diffusion Model Alignment Using Direct Preference Optimization (2023, methodology): “Methods that fine-tune the policy model directly on feedback data [2, 12], or utilize a ranking loss on preference data to directly train the policy model [36, 52, 60, 62] have emerged.”
- From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning (2023, methodology): “A well-formulated instruction or prompt provides essential contextual information, refining the model’s capability to generate relevant and task-specific outputs (Taori et al., 2023; Ouyang et al., 2022).”
- CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation (2024, neutral): “Further implementation details and filtering statistics are provided in Ap-pendices A and D, respectively.”
- Knowledge Graph Prompting for Multi-Document Question Answering (2023, methodology): “As the predicted answer may not overlap with the ground-truth one, we additionally check the correctness of the answer following (Liu et al. 2023b; Dubois et al. 2023; Zhang et al. 2023) by prompting the LLM.”
- VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, methodology): “Beyond existing metrics and baselines, following the recent line of work utilizing API-accessed LLMs with a prompt for automatic evaluation [6, 50], we consider two GPT-4[7] backed evaluation metrics.”