Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models

By Avi Singh, John D. Co-Reyes, Rishabh Agarwal, Ankesh Anand, Piyush Patil, Peter J. Liu, J. Harrison, Jaehoon Lee, Kelvin Xu, Aaron Parisi and 28 more. Trans. Mach. Learn. Res., 2023

337 citations recorded by the source corpus.

Fine-tuning language models~(LMs) on human-generated data remains a prevalent practice. However, the performance of such models is often limited by the quantity and diversity of high-quality human data. In this paper, we explore whether we can go beyond human data on tasks where we have access to scalar feedback, for example, on math problems where one can verify correctness. To do so, we investigate a simple self-training method based on expectation-maximization, which we call ReST$^{EM}$, where we (1) generate samples from the model and filter them using binary feedback, (2) fine-tune the model on these samples, and (3) repeat this process a few times. Testing on advanced MATH reasoning and APPS coding benchmarks using PaLM-2 models, we find that ReST$^{EM}$ scales favorably with model size and significantly surpasses fine-tuning only on human data. Overall, our findings suggest self-training with feedback can substantially reduce dependence on human-generated data.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/48362b169a235ca650918c489c8cea4c597da645

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Training Language Models to Self-Correct via Reinforcement Learning (2024, positive): “Variants of this approach have been shown to scale well on single-turn reasoning problems (Singh et al., 2023; Zelikman et al., 2022).”
LoRA Learns Less and Forgets Less (2024, methodology): “Singh et al. (2023) iterate over this procedure multiple times (“Self-training”) using an expectation-maximization approach.”
Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, background): “…to the broader goal of converting weak models to strong models without the requirement for extra training data, which is of central interest in machine learning that can be traced back to the boosting algorithms (Kearns & Valiant, 1994; Schapire, 1990; Freund, 1995; Freund & Schapire, 1997).”
Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters (2024, methodology): “…32, 39]; 2) improving the LLM proposal distribution by either applying targeted optimization on specific reasoning tasks by finetuning with RL [32, 35, 49, 50] enabling models to critique and revise their answers iteratively [4, 8, 23, 30]; 3) enabling LLMs to benefit from additional test-time…”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “…al., 2023; Yuan et al., 2024a; Xu et al., 2023b; Zelikman et al., 2022; Chen et al., 2024b; Zheng et al., 2024; Li et al., 2024c; Zhao et al., 2024; Singh et al., 2023; Chen et al., 2024c; Hosseini et al., 2024) A notable example of this methodology is Self-Instruct (Wang et al., 2022a), which…”
Self-Training: A Survey (2022, positive): “A recent innovative approach, called Reinforced self-training (ReST) has emerged, particularly notable for its application in conditional language modeling [31, 74].”
Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvement (2024, methodology): “For the annotated problems, we utilize an online Rejection Fine-Tuning (RFT) (Yuan et al., 2023; Singh et al., 2024) approach to iteratively generate tool-integrated reasoning paths whose final answers align with the reference answers.”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, positive): “Following Singh et al. (2023), we then sub-sample the CoTs to 10 random unique CoTs per question to balance difficulties of questions.”
SimpleRL-Zoo: Investigating and Taming Zero Reinforcement Learning for Open Base Models in the Wild (2025, neutral): “Enforcing rigid format reward (e.g., enclosing answers within boxes) (DeepSeek-AI et al., 2025a) significantly penalizes exploration (Singh et al., 2023; Wang et al., 2024), particularly for base models that initially struggle with instruction following.”
Generative Verifiers: Reward Modeling as Next-Token Prediction (2024, neutral): “One naive approach is to simply use the ‘Let’s verify step by step’ prompt given a problem-solution pair, and keep the generated rationales only when they accurately verify the correctness of a solution (Singh et al., 2023; Zelikman et al., 2022).”