SimPO: Simple Preference Optimization with a Reference-Free Reward

By Yu Meng, Mengzhou Xia, Danqi Chen. Neural Information Processing Systems, 2024

1,086 citations recorded by the source corpus.

Direct Preference Optimization (DPO) is a widely used offline preference optimization algorithm that reparameterizes reward functions in reinforcement learning from human feedback (RLHF) to enhance simplicity and training stability. In this work, we propose SimPO, a simpler yet more effective approach. The effectiveness of SimPO is attributed to a key design: using the average log probability of a sequence as the implicit reward. This reward formulation better aligns with model generation and eliminates the need for a reference model, making it more compute and memory efficient. Additionally, we introduce a target reward margin to the Bradley-Terry objective to encourage a larger margin between the winning and losing responses, further improving the algorithm's performance. We compare SimPO to DPO and its latest variants across various state-of-the-art training setups, including both base and instruction-tuned models such as Mistral, Llama 3, and Gemma 2. We evaluate on extensive chat-based evaluation benchmarks, including AlpacaEval 2, MT-Bench, and Arena-Hard. Our results demonstrate that SimPO consistently and significantly outperforms existing approaches without substantially increasing response length. Specifically, SimPO outperforms DPO by up to 6.4 points on AlpacaEval 2 and by up to 7.5 points on Arena-Hard. Our top-performing model, built on Gemma-2-9B-it, achieves a 72.4% length-controlled win rate on AlpacaEval 2, a 59.1% win rate on Arena-Hard, and ranks 1st on Chatbot Arena among<10B models with real user votes.

Subfields: Automotive Engineering, Computer Networks and Communications, Management Science and Operations Research

View Paper: https://www.semanticscholar.org/paper/c3f1fae241a3c2449e675ab750873d800f95513c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Networks and Communications

In Management Science and Operations Research

Citations in the corpus

TÜLU 3: Pushing Frontiers in Open Language Model Post-Training (2024, positive): “We use the length-normalized variant of DPO proposed in Meng et al. [2024].”
Beyond the 80/20 Rule: High-Entropy Minority Tokens Drive Effective Reinforcement Learning for LLM Reasoning (2025, positive): “Offline meth-ods like DPO (Rafailov et al., 2023), SimPO (Meng et al., 2024), and KTO (Ethayarajh et al., 2024) optimize policies using pre-collected preferences, typically from human annotators or LLMs.”
Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning (2025, positive): “To simplify RL-based tuning, direct optimization methods such as Direct Preference Optimization (DPO) (Rafailov et al., 2023) and SimPO (Meng et al., 2024) have been proposed.”
ToolRL: Reward is All Tool Learning Needs (2025, positive): “GRPO’s flexible reward function enables adaptation to diverse objectives, such as assigning weights to sub-tasks (Yu et al., 2024) or constraining tool use frequency (Li et al., 2025b).”
Stop Overthinking: A Survey on Efficient Reasoning for Large Language Models (2025, positive): “Without relying on a reference model, DAST [74] employs SimPO [60] to fine-tune reasoning LLMs using a constructed length-preference dataset.”
Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (2024, methodology): “To explore this, we include evaluations to measure the bias across different LLM architectures and scales, following Zheng et al. (2024) and Meng et al. (2024).”
Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMs (2024, positive): “This has been expanded with methods like debating, self-correction, self-critique, and plan-and-solve (Liang et al., 2024; Du et al., 2024; Xiong et al., 2023; Kumar et al., 2024; Kamoi et al., 2024; Ke et al., 2023; Lin et al., 2024; Yu et al., 2024; Wang et al., 2023a).”
WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct (2023, positive): “NLP models face challenges with complex reasoning, including mathematical (Long et al., 2024; Zhang et al., 2024b; Xia et al., 2024), common-sense (Talmor et al., 2019).”
Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing (2024, methodology): “Specifically, we follow [39] and use the models fine-tuned with the UltraChat dataset (for instruction tuning) and Ultrafeedback dataset (for preference optimization) [13].”
WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wild (2024, methodology): “In Figure 4, we select a set of popular models for analysis: Llama-3-8B-Inst [22], Llama-3-8B-Inst-SimPO [20], Yi-1.”
S-SPPO: Semantic-Calibrated Self-Play Preference Optimization (2026, unknown): “IPO (Azar et al., 2024) Meng et al., 2024) introduces a reference-free objective that mitigates length bias by maximizing the gap between the preferred and dispreferred responses’ average log-likelihood.”
UniSD: Towards a Unified Self-Distillation Framework for Large Language Models (2026, unknown): “In practice, adaptation pipelines often rely on stronger external models for supervision, including synthetic data generation [1–3], reinforcement learning [4, 5], and distillation from stronger teacher models [6, 7].”