What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning

By Wei Liu, Weihao Zeng, Keqing He, Yong Jiang, Junxian He. International Conference on Learning Representations, 2023

405 citations recorded by the source corpus.

Instruction tuning is a standard technique employed to align large language models to end tasks and user preferences after the initial pretraining phase. Recent research indicates the critical role of data engineering in instruction tuning -- when appropriately selected, only limited data is necessary to achieve superior performance. However, we still lack a principled understanding of what makes good instruction tuning data for alignment, and how we should select data automatically and effectively. In this work, we delve deeply into automatic data selection strategies for alignment. We start with controlled studies to measure data across three dimensions: complexity, quality, and diversity, along which we examine existing methods and introduce novel techniques for enhanced data measurement. Subsequently, we propose a simple strategy to select data samples based on the measurement. We present deita (short for Data-Efficient Instruction Tuning for Alignment), a series of models fine-tuned from LLaMA and Mistral models using data samples automatically selected with our proposed approach. Empirically, deita performs better or on par with the state-of-the-art open-source alignment models with only 6K SFT training data samples -- over 10x less than the data used in the baselines. When further trained with direct preference optimization (DPO), deita-Mistral-7B + DPO trained with 6K SFT and 10K DPO samples achieve 7.55 MT-Bench and 90.06% AlpacaEval scores. We anticipate this work to provide tools on automatic data selection, facilitating data-efficient alignment. We release our models as well as the selected datasets for future researches to effectively align models more efficiently.

Subfields: Artificial Intelligence, Information Systems

View Paper: https://www.semanticscholar.org/paper/41113411e1748a34bb80f12c761b7af1ed6dbb90

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graph (2023, neutral): “Using the beam search algorithm (Jurafsky & Martin, 2009) in KG/LLM reasoning (Atif et al., 2023; Sun et al., 2023a; Xie et al., 2023; Liu et al., 2024), ToG allows LLM to dynamically explore a number of reasoning paths in KG and make decisions accordingly.”
Yi: Open Foundation Models by 01.AI (2024, methodology): “In contrast, our method aligns with the LIMA [94] and DEITA [48] approach, which focus on data selection rather than scaling.”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, background): “There are also mixture datasets (e.g., [24, 49, 38, 67]) that combine or select high-quality instruction data from various existing open-source instruction datasets to enhance coverage [24, 49] and/or improve overall performance [38, 67].”
LESS: Selecting Influential Data for Targeted Instruction Tuning (2024, background): “Increasing data quality and diversity instead of quantity has been shown to more effectively induce instruction following abilities (Cao et al., 2023; Chen et al., 2023a; Bukharin & Zhao, 2023; Du et al., 2023; Liu et al., 2023).”
How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition (2023, neutral): “1”