LIMA: Less Is More for Alignment
By Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, L. Yu and 5 more. 2023
1,362 citations recorded by the source corpus.
Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.
Subfields: Health Information Management, History and Philosophy of Science, Information Systems, Large Language Models, Alignment & RLHF
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Health Information Management
In History and Philosophy of Science
- Adrian Weller
- Yejin Choi
- Dan Jurafsky
- Chandra Bhagavatula
- Tatsunori B. Hashimoto
- Jack Hessel
- Abhilasha Ravichander
- Erik Cambria
In Information Systems
- Pengfei Liu
- Graham Neubig
- Weinan Zhang
- Dan Jurafsky
- Mirac Suzgun
- Bill Yuchen Lin
- Jun Wang
- Weizhe Yuan
In Large Language Models
In Alignment & RLHF
- Dylan Hadfield-Menell
- David Krueger
- Yejin Choi
- Dan Jurafsky
- J'er'emy Scheurer
- Tatsunori B. Hashimoto
- Anca Dragan
- Tomasz Korbak
Citations in the corpus
- Improved Baselines with Visual Instruction Tuning (2023, background): “In NLP, studies show that the quality of instruction-following data largely affects the capability of the resulting instruction-following models [48].”
- RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (2023, neutral): “It contains context, response generated by LLM with and span-level annotation. tual or accurate information (Rawte et al., 2023).”
- VisIT-Bench: A Benchmark for Vision-Language Instruction Following Inspired by Real-World Use (2023, background): “As the downstream capabilities of these models are influenced by the quality of the training dataset, there has also been extensive work on developing instruction-following datasets [39, 56, 57, 13, 58] .”
- Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model (2024, background): “This is even more difficult for multilingual data and most efforts to date have focused primarily on English datasets (Köpf et al., 2023; Conover et al., 2023; Zhou et al., 2023).”
- How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, methodology): “, 2023) while diverse and complex human instructions are applauded for aligning human intentions (Wang et al., 2023c; Taori et al., 2023; Xu et al., 2023; Zhou et al., 2023; Wang et al., 2023a; Lu et al., 2023).”
- Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation (2023, methodology): “In response to this challenge, LLM providers have implemented a range of training techniques aimed at “align-ing” these models with human values before releasing them (Ouyang et al., 2022; Bai et al., 2022a; Korbak et al., 2023; Zhou et al., 2023).”
- Large Language Models are not Fair Evaluators (2023, background): “Recently, a lot of works (Zheng et al., 2023; Wang et al., 2023a; Sun et al., 2023; Zhou et al., 2023) have been utilizing LLMs such as GPT-4 as evaluators to compare the performance of two AI assistants.”
- Explainability for Large Language Models: A Survey (2023, background): “(Zhou et al., 2023) investigated the relative importance of pre-training versus instruction fine-tuning in language models.”
- MAmmoTH: Building Math Generalist Models through Hybrid Instruction Tuning (2023, methodology): “However, most existing work (Zhou et al., 2023a) in PoT is limited to proprietary models like GPT-4 (OpenAI, 2023) and Codex (Chen et al., 2021).”
- A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, neutral): “Pre-training is generally considered a crucial stage for LLM to acquire knowledge and skills (Zhou et al., 2023a).”
- Instruction Tuning for Large Language Models: A Survey (2023, neutral): “For automatic evaluation, which is conducted by asking GPT-4 to rate responses and a higher rate score denotes better performance, LIMA outperforms InstructGPT and Alpaca by 20% and 36%, respectively, achieving comparable results to BARD, while underperforming Claude and GPT-4.”
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023, methodology): “Inspired from (Zhou et al., 2023), we feed the prompt in appendix F to ChatGPT to score the model’s outputs based on response quality, with scores ranging from 1 to 5.”