Reinforced Self-Training (ReST) for Language Modeling

By Caglar Gulcehre, T. Paine, S. Srinivasan, Ksenia Konyushkova, L. Weerts, Aditya Siddhant, Alexa Ahern, Miaosen Wang, Chenjie Gu, Wolfgang Macherey and 3 more. arXiv.org, 2023

473 citations recorded by the source corpus.

Reinforcement learning from human feedback (RLHF) can improve the quality of large language model's (LLM) outputs by aligning them with human preferences. We propose a simple algorithm for aligning LLMs with human preferences inspired by growing batch reinforcement learning (RL), which we call Reinforced Self-Training (ReST). Given an initial LLM policy, ReST produces a dataset by generating samples from the policy, which are then used to improve the LLM policy using offline RL algorithms. ReST is more efficient than typical online RLHF methods because the training dataset is produced offline, which allows data reuse. While ReST is a general approach applicable to all generative learning settings, we focus on its application to machine translation. Our results show that ReST can substantially improve translation quality, as measured by automated metrics and human evaluation on machine translation benchmarks in a compute and sample-efficient manner.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/182c7b40ff7560a5545764814338f55a2098e441

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

ORPO: Monolithic Preference Optimization without Reference Model (2024, methodology): “Alignment with Supervised Fine-tuning Preference alignment methods in reinforcement learning (RL) often leverage supervised fine-tuning (SFT) to ensure the stable update of the active policy in relation to the old policy (Schulman et al., 2017).”
Alphazero-like Tree-Search can Guide Large Language Model Decoding and Training (2023, unknown): “…al., 2023; Feng et al., 2023; Taylor et al., 2022), more sophisticated prompt design (Wei et al., 2022; Zhou et al., 2022; Creswell et al., 2022), or better training algorithms with Supervised Learning or Reinforcement Learning (RL) (Dong et al., 2023; Gulcehre et al., 2023; Rafailov et al., 2023).”
Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Models (2023, methodology): “…(Section 4), has demonstrated success in enhancing language models across diverse domains, including machine translation (Norouzi et al., 2016; Gulcehre et al., 2023), semantic parsing (Agarwal et al., 2019), preference alignment (Dong et al., 2023), and elementary reasoning (Zelikman et al.,…”
Making Large Language Models Better Reasoners with Alignment (2023, unknown): “However, RL-based methods face limitations concerning training efficiency and complexity; 2) Rejective Sampling (Touvron et al., 2023; Gulcehre et al., 2023; Dong et al., 2023; Liu et al., 2023), which aligns large language models by sampling multiple responses and choosing high-quality ones to…”
Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking (2024, background): “For example, works explore this in the context of multihop question answering (Zhao et al., 2023), math (Luo et al., 2023; Uesato et al., 2022), machine translation (Gulcehre et al., 2023).”
Diffusion Model Alignment Using Direct Preference Optimization (2023, background): “Alternative approaches sample base model answers and select based on predicted rewards [4, 6, 16] to use for supervised training [3, 18, 53].”
A Survey on Knowledge Distillation of Large Language Models (2024, methodology): “This is Gulcehre et al. (2023) introduces a Reinforced Self-Training (ReST) framework that cyclically alternates between Grow and Improve stages to progressively obtain better self-knowledge and refine the student model.”
Generative Verifiers: Reward Modeling as Next-Token Prediction (2024, unknown): “Discriminative RMs We finetune Gemma-based discriminative RMs by using a special token’s logit for classification.”
Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? (2025, unknown): “This stage typically falls into three main categories: supervised fine-tuning using human-curated or distilled data (Wang et al., 2023), self-improvement iteration (Zelikman et al., 2022; Gulcehre et al., 2023), and reinforcement learning (Ouyang et al., 2022).”
Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-constraint (2023, methodology): “The multi-step approximation in Algorithm 3 shares similar spirit with the iterative framework (i.e., the RAFT algorithm) proposed in Dong et al. (2023) and was also considered in Touvron et al. (2023) and Gulcehre et al. (2023).”
Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts (2024, methodology): “These ideas also extend to other approaches, such as rejection sampling fine-tuning [Dong et al., 2023; Gulcehre et al., 2023] and iterative direct preference learning [Xiong et al., 2023; Guo et al., 2024; Xie et al., 2024].”
ODIN: Disentangled Reward Mitigates Hacking in RLHF (2024, background): “Many offline alternatives have also shown promises for better learning from feedbacks, which includes SLiC-HF [Zhao et al., 2023], DPO [Rafailov et al., 2023], IPO [Azar et al., 2023], KTO [Ethayarajh et al., 2023], ReST [Gulcehre et al., 2023] and RSO [Liu et al., 2024].”
ReFT: Reasoning with Reinforced Fine-Tuning (2024, methodology): “We can see this simple baseline can sometimes improve the performance compared with SFT (He et al., 2020; Gulcehre et al., 2023) but the improvements are far behind the one made by ReFT.”
Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, unknown): “The primary distinction between ReST and other RL-based methods including ours is that ReST iteratively refines the model by fitting the best response sampled from the current model, without applying negative gradients to penalize incorrect responses.”
Self-Training: A Survey (2022, background): “In this case, the understanding of the anomaly patterns and characteristics speciﬁc to the domain are incorporated in the model.”
UltraFeedback: Boosting Language Models with High-quality Feedback (2023, unknown): “, 2023b), design novel algorithms (Gulcehre et al., 2023; Rafailov et al., 2023; Song et al., 2023a) to utilize preference data more efficiently, or align LLMs with various types of feedbacks (Yao et al.”
Self-Rewarding Language Models (2024, unknown): “Reinforced Self-Training (ReST) [Gulcehre et al., 2023] uses a fixed, external reward to curate new high-quality examples to iteratively add to the training set, improving performance.”
RLHF Workflow: From Reward Modeling to Online RLHF (2024, background): “• Rejection Sampling is another popular ensemble-based exploration approach (Nakano et al., 2021; Dong et al., 2023; Gulcehre et al., 2023).”