How Abilities in Large Language Models are Affected by Supervised Fine-tuning Data Composition

By Guanting Dong, Hongyi Yuan, Keming Lu, Chengpeng Li, Mingfeng Xue, Dayiheng Liu, Wei Wang, Zheng Yuan, Chang Zhou, Jingren Zhou. Annual Meeting of the Association for Computational Linguistics, 2023

310 citations recorded by the source corpus.

Large language models (LLMs) with enormous pre-training tokens and parameters emerge diverse abilities, including math reasoning, code generation, and instruction following. These abilities are further enhanced by supervised fine-tuning (SFT). While the open-source community has explored ad-hoc SFT for enhancing individual capabilities, proprietary LLMs exhibit versatility across various skills. Therefore, understanding the facilitation of multiple abilities via SFT is paramount. In this study, we specifically focuses on the interplay of data composition between mathematical reasoning, code generation, and general human-aligning abilities during SFT. We propose four intriguing research questions to explore the association between model performance and various factors including data amount, composition ratio, model size and SFT strategies. Our experiments reveal that distinct capabilities scale differently and larger models generally show superior performance with same amount of data. Mathematical reasoning and code generation consistently improve with increasing data amount, whereas general abilities plateau after roughly a thousand samples. Moreover, we observe data composition appears to enhance various abilities under limited data conditions, yet can lead to performance conflicts when data is plentiful. Our findings also suggest the amount of composition data influences performance more than the composition ratio. In analysis of SFT strategies, we find that sequentially learning multiple skills risks catastrophic forgetting. Our proposed Dual-stage Mixed Fine-tuning (DMT) strategy offers a promising solution to learn multiple abilities with different scaling patterns.

Subfields: Artificial Intelligence, Materials Chemistry

View Paper: https://www.semanticscholar.org/paper/5088a04d1a9f42b967f3dcf791145e8aa367fc54

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Fei Huang
2. Binyuan Hui
3. Jingren Zhou
4. Sinan Tan
5. Xiaohuan Zhou
6. Ge Zhang
7. Wenhao Huang
8. Shuai Bai

In Materials Chemistry

1. Fei Huang
2. Junyang Lin

Citations in the corpus

• Yi: Open Foundation Models by 01.AI (2024, methodology): “Motivated by Dong et al. [20], this process involved experimenting with {1, 1/2, 1/4, 1/8, 1/16, 1/32, 1/64} proportions for each ability.”
• Search-o1: Agentic Search-Enhanced Large Reasoning Models (2025, unknown): “…redundant information, directly inputting them into LRMs may disrupt the original coherence of reasoning and even introduce noise [62, 72, 26]. disrupt the original coherence of reasoning introduce noise (Wu et al., 2024; Yoran 2 Related Work (2) Limited Ability to Understand Long Documents.”
• Qwen2 Technical Report (2024, methodology): “Based on these criteria, we select a set of representative instructions (Dong et al., 2023).”
• ORPO: Monolithic Preference Optimization without Reference Model (2024, background): “SFT plays a significant role in tailoring the pre-trained language models to the desired domain (Zhou et al., 2023a; Dong et al., 2024) by increasing the log probabilities of pertinent tokens.”
• Sparsity-Accelerated Training for Large Language Models (2024, background): “2) supervised fine-tuing (SFT) on labeled data to enhance the capacity of LLMs to follow human instructions (Ouyang et al., 2022; Dong et al., 2023; Ansell et al., 2024).”