SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training

By Tianzhe Chu, Yuexiang Zhai, Jihan Yang, Shengbang Tong, Saining Xie, Dale Schuurmans, Quoc V. Le, Sergey Levine, Yi Ma. International Conference on Machine Learning, 2025

649 citations recorded by the source corpus.

Supervised fine-tuning (SFT) and reinforcement learning (RL) are widely used post-training techniques for foundation models. However, their roles in enhancing model generalization capabilities remain unclear. This paper studies the difference between SFT and RL on generalization and memorization, focusing on text-based rule variants and visual variants. We introduce GeneralPoints, an arithmetic reasoning card game, and adopt V-IRL, a real-world navigation environment, to assess how models trained with SFT and RL generalize to unseen variants in both textual and visual domains. We show that RL, especially when trained with an outcome-based reward, generalizes across both rule-based textual and visual variants. SFT, in contrast, tends to memorize training data and struggles to generalize out-of-distribution scenarios. Further analysis reveals that RL improves the model's underlying visual recognition capabilities, contributing to its enhanced generalization in the visual domain. Despite RL's superior generalization, we show that SFT remains essential for effective RL training; SFT stabilizes the model's output format, enabling subsequent RL to achieve its performance gains. These findings demonstrates the capability of RL for acquiring generalizable knowledge in complex, multi-modal tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/a99dee9602e21a71526b9681d8dba37c55b66941

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Chujie Zheng
2. Andrew Zhao
3. Kai Dang
4. Le Yu
5. Tiancheng Zhao
6. Jianfeng Gao
7. Maxine Eskénazi
8. Minlie Huang

In Computer Vision and Pattern Recognition

1. Chujie Zheng
2. Kai Dang
3. Tiancheng Zhao

Citations in the corpus

• Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (2025, positive): “5 (Team et al., 2025), and Qwen3 (Yang et al., 2025) have aimed to match or exceed its performance.”
• VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model (2025, positive): “These findings strongly support the conclusion proposed by [12]: “SFT Memorizes, RL Generalizes” .”
• Video-R1: Reinforcing Video Reasoning in MLLMs (2025, neutral): “…reasoning, they provide a broader assessment of the model’s overall video comprehension abilities, and include subsets that require temporal or logical reasoning. and TempCompass, performance even slightly drops after SFT, likely due to overfitting or limited generalization in unseen scenarios [3].”
• A^2Agent: Action-Aware Reinforcement Learning for Repository-Level Code Localization Agents (2026, unknown): “Early agentic training relied primarily on SFT to imitate teacher trajectories, but it struggles to generalize to unfamiliar codebases at inference time (Deng et al., 2025; Chu et al., 2025).”