Buy 4 REINFORCE Samples, Get a Baseline for Free!

By Wouter Kool, Herke Van Hoof, Max Welling. 2019

262 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Understanding R1-Zero-Like Training: A Critical Perspective (2025, unknown): “We note that ˜ A i , t computed above is equivalent to that of REINFORCE Leave-One-Out (RLOO) (Ahmadian et al., 2024; Kool et al., 2019) up to a scaling factor, which can be subsumed into the learning rate without affecting the RL dynamics.”
Gradient Estimation with Stochastic Softmax Tricks (2020, methodology): “[43] Wouter Kool, Herke van Hoof, and Max Welling.”
Simplified and Generalized Masked Diffusion for Discrete Data (2024, unknown): “To fix this, we used the REINFORCE leave-one-out estimator [53, 54] to compute low-variance unbiased gradients for optimizing w .”
POMO: Policy Optimization with Multiple Optima for Reinforcement Learning (2020, background): “Note that these trajectories are fundamentally different from repeatedly sampled N trajectories under the START token scheme [28].”
Process Reinforcement through Implicit Rewards (2025, unknown): “Different choices of b include, e.g. directly using values (Williams, 1992), group average of rewards (Shao et al., 2024), and leave-one-out average of rewards (Ahmadian et al., 2024; Kool et al., 2019).”
The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models (2025, unknown): “To reduce variance, GRPO (Shao et al., 2024) and RLOO (Kool et al., 2019; Ahmadian et al., 2024) further incorporates group-wise normalization.”
DIFUSCO: Graph-based Diffusion Solvers for Combinatorial Optimization (2023, background): “…years since [22, 64, 90, 25, 68], including better network backbones [106, 64, 9]), more advanced deep reinforcement learning algorithms [57, 78, 65, 67, 87, 114, 109, 18], improved training strategies [59, 7], and for a wider range of NPC problems such as Capacitated Vehicle Routing Problem…”
Group-in-Group Policy Optimization for LLM Agent Training (2025, unknown): “Going further, LOOP [48] introduces a hybrid method combining REINFORCE leave-one-out (RLOO) [15, 16] with PPO-style updates, achieving state-of-the-art results in AppWorld [49].”
Kimi k1.5: Scaling Reinforcement Learning with LLMs (2025, unknown): “For each problem x , k responses are sampled using the reference policy π θ i , and the gradient is given by To those familiar with policy gradient methods, this gradient resembles the policy gradient of (2) using the mean of sampled rewards as the baseline (Kool et al. 2019; Ahmadian et al. 2024).”