LIMA: Less Is More for Alignment

By Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, L. Yu and 5 more. 2023

1,362 citations recorded by the source corpus.

Large language models are trained in two stages: (1) unsupervised pretraining from raw text, to learn general-purpose representations, and (2) large scale instruction tuning and reinforcement learning, to better align to end tasks and user preferences. We measure the relative importance of these two stages by training LIMA, a 65B parameter LLaMa language model fine-tuned with the standard supervised loss on only 1,000 carefully curated prompts and responses, without any reinforcement learning or human preference modeling. LIMA demonstrates remarkably strong performance, learning to follow specific response formats from only a handful of examples in the training data, including complex queries that range from planning trip itineraries to speculating about alternate history. Moreover, the model tends to generalize well to unseen tasks that did not appear in the training data. In a controlled human study, responses from LIMA are either equivalent or strictly preferred to GPT-4 in 43% of cases; this statistic is as high as 58% when compared to Bard and 65% versus DaVinci003, which was trained with human feedback. Taken together, these results strongly suggest that almost all knowledge in large language models is learned during pretraining, and only limited instruction tuning data is necessary to teach models to produce high quality output.

Subfields: Health Information Management, History and Philosophy of Science, Information Systems, Large Language Models, Alignment & RLHF

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Health Information Management

  1. Prateek Mittal

In History and Philosophy of Science

  1. Adrian Weller
  2. Yejin Choi
  3. Dan Jurafsky
  4. Chandra Bhagavatula
  5. Tatsunori B. Hashimoto
  6. Jack Hessel
  7. Abhilasha Ravichander
  8. Erik Cambria

In Information Systems

  1. Pengfei Liu
  2. Graham Neubig
  3. Weinan Zhang
  4. Dan Jurafsky
  5. Mirac Suzgun
  6. Bill Yuchen Lin
  7. Jun Wang
  8. Weizhe Yuan

In Large Language Models

  1. Qun Liu
  2. Xin Jiang
  3. Yejin Choi
  4. Graham Neubig
  5. Yanwei Li
  6. Chengyao Wang
  7. Lifeng Shang
  8. Wenhu Chen

In Alignment & RLHF

  1. Dylan Hadfield-Menell
  2. David Krueger
  3. Yejin Choi
  4. Dan Jurafsky
  5. J'er'emy Scheurer
  6. Tatsunori B. Hashimoto
  7. Anca Dragan
  8. Tomasz Korbak

Citations in the corpus