Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models

By Zixiang Chen, Yihe Deng, Huizhuo Yuan, Kaixuan Ji, Quanquan Gu. International Conference on Machine Learning, 2024

604 citations recorded by the source corpus.

Harnessing the power of human-annotated data through Supervised Fine-Tuning (SFT) is pivotal for advancing Large Language Models (LLMs). In this paper, we delve into the prospect of growing a strong LLM out of a weak one without the need for acquiring additional human-annotated data. We propose a new fine-tuning method called Self-Play fIne-tuNing (SPIN), which starts from a supervised fine-tuned model. At the heart of SPIN lies a self-play mechanism, where the LLM refines its capability by playing against instances of itself. More specifically, the LLM generates its own training data from its previous iterations, refining its policy by discerning these self-generated responses from those obtained from human-annotated data. Our method progressively elevates the LLM from a nascent model to a formidable one, unlocking the full potential of human-annotated demonstration data for SFT. Theoretically, we prove that the global optimum to the training objective function of our method is achieved only when the LLM policy aligns with the target data distribution. Empirically, we evaluate our method on several benchmark datasets including the HuggingFace Open LLM Leaderboard, MT-Bench, and datasets from Big-Bench. Our results show that SPIN can significantly improve the LLM's performance across a variety of benchmarks and even outperform models trained through direct preference optimization (DPO) supplemented with extra GPT-4 preference data. This sheds light on the promise of self-play, enabling the achievement of human-level performance in LLMs without the need for expert opponents. Codes are available at https://github.com/uclaml/SPIN.

View Paper: https://www.semanticscholar.org/paper/ef9e058d22d190fdd38ddee367cf6aa8d1a14bd5

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

R1-VL: Learning to Reason with Multimodal Large Language Models via Step-Wise Group Relative Policy Optimization (2025, unknown): “Recently, RL is increasingly adopted to enhance LLMs’ reasoning capabilities [4, 6, 12, 22, 32, 45], especially for mathematical problem solving.”
Instruction Tuning for Large Language Models: A Survey (2023, unknown): “Despite these risks, there remains effective work in the area of self-improvement: [20], standing for Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models, represents a specialized approach to self-improvement centered around a self-play mechanism.”
Training Language Models to Self-Correct via Reinforcement Learning (2024, unknown): “As x increases from 0 to ∞ , the value of ( − 58 x oscillates between positive and negative values, but its absolute value decreases.”
Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study (2024, methodology): “Among these reward-free methods, DPO (Rafailov et al., 2023) has demonstrated strong performances and become popular in the community (MistralAI, 2023; Chen et al., 2024; Yuan et al., 2024).”
LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models (2024, unknown): “(3) Exploring stronger fine-tuning methods for conversational models, e . g ., self-play (Chen et al., 2024b; Yuan et al., 2024).”
Model Alignment as Prospect Theoretic Optimization (2024, background): “Online Alignment A recent string of work has centered on the idea of “self-training” or “self-play”, during which the policy is continually aligned on online data sampled from itself and then filtered (Chen et al., 2024; Yuan et al., 2024).”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “More recently, Ji et al. (2024) introduce Aligner, a novel approach for aligning LLMs with human values and intentions by utilizing weak supervisory signals from smaller models to improve the performance of larger models.”
Self-Rewarding Language Models (2024, unknown): “Head to head performance results are provided in Figure 3.”
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “Concurrently, various works have explored alternative game-theoretic formulations (Wang et al., 2025; P´asztor et al., 2025; Tang et al., 2025) or autonomous self-improvement mechanisms (Ji et al., 2024; Chen et al., 2024; Le et al., 2025; Xiang et al., 2025) to enhance stability.”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, background): “5-turbo-1106, which are aligned by DPO or PPO and are much larger than our base model.”
Binary Classifier Optimization for Large Language Model Alignment (2024, unknown): “To reduce the effort required to collect preference datasets, methodologies have been proposed that either let the LLM itself perform comparisons of completions (Yuan et al., 2024) or treat the LLM’s completions as rejected completions (Chen et al., 2024b) .”