The Mirage of Action-Dependent Baselines in Reinforcement Learning

By George Tucker, Surya Bhupatiraju, Shixiang Shane Gu, Richard E. Turner, Zoubin Ghahramani, Sergey Levine. 2018

145 citations recorded by the source corpus.

Policy gradient methods are a widely used class of model-free reinforcement learning algorithms where a state-dependent baseline is used to reduce gradient estimator variance. Several recent papers extend the baseline to depend on both the state and action and suggest that this significantly reduces variance and improves sample efficiency without introducing bias into the gradient estimates. To better understand this development, we decompose the variance of the policy gradient estimator and numerically show that learned state-action-dependent baselines do not in fact reduce variance over a state-dependent baseline in commonly tested benchmark domains. We confirm this unexpected result by reviewing the open-source code accompanying these prior papers, and show that subtle implementation decisions cause deviations from the methods presented in the papers and explain the source of the previously observed empirical gains. Furthermore, the variance decomposition highlights areas for improvement, which we demonstrate by illustrating a simple change to the typical value function parameterization that can significantly improve performance.

Subfields: Artificial Intelligence, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

Citations in the corpus

SNAS: Stochastic Neural Architecture Search (2018, background): “To assign credits to actions both temporally and laterally is an important topic in reinforcement learning (Precup, 2000; Schulman et al., 2015; Tucker et al., 2018; Xu et al., 2018).”
The Surprising Effectiveness of PPO in Cooperative Multi-Agent Games (2021, positive): “Our empirical analysis of PPO’s implementation and hyperparameter factors in multi-agent settings is similar to the studies of policy-gradient methods in single-agent RL [33, 17, 9, 1].”
From Importance Sampling to Doubly Robust Policy Gradient (2019, neutral): “The connection between DR and PG was lightly touched by Tucker et al. (2018), although the authors’ main goal was to challenge the success of state-action-dependent baseline methods in benchmarks, and did not give a more detailed analysis on this connection.”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “[23] empirically analyzed the three variance components in (8) in LQG and simulated robot locomotion tasks.”
Reward-estimation variance elimination in sequential decision processes (2018, methodology): “The limitation of AB is that it takes care of the variance due to an immediate action a t , while leaving the variance originating from the rest of the trajectory mostly unchecked [18].”
A Minimalist Approach to Offline Reinforcement Learning (2021, methodology): “…run times of the algorithms, and make causal attributions of performance gains and transfers of techniques across algorithms difficult, as in the case for many online RL algorithms [Henderson et al., 2017, Tucker et al., 2018, Engstrom et al., 2020, Andrychowicz et al., 2021, Furuta et al., 2021].”
What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study (2020, background): “[26] showed that the gains, which had been attributed to one of the recently proposed policy gradients improvements, were, in fact, caused by the implementation details.”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “A number of implementations of RL algorithms such as Q-Prop and IPG often have subtle, unintentional biases (Tucker et al., 2018).”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “[59] found that several published works on action-dependant baselines contained bugs and errors —”
Implementation Matters in Deep RL: A Case Study on PPO and TRPO (2020, background): “, 2018), hard to reproduce (Henderson et al., 2017; Tucker et al., 2018), unreliable across runs (Henderson et al.”
Policy Optimization via Importance Sampling (2018, background): “Baselines are typically constant (REINFORCE), time-dependent (G(PO)MDP) or state-dependent (actor-critic), but these approaches have been recently extended to account for action-dependent baselines [57, 64].”
Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPO (2020, background): “…research found the existing deep RL methods to be brittle (Henderson et al., 2017; Zhang et al., 2018), hard to reproduce (Hen-derson et al., 2017; Tucker et al., 2018), unreliable across runs (Henderson et al., 2017; 2018), and sometimes outperformed by simple baselines (Mania et al., 2018).”
What Matters for On-Policy Deep Actor-Critic Methods? A Large-Scale Study (2021, background): “Tucker et al. [26] showed that the gains, which had been attributed to one of the recently proposed policy gradients improvements, were, in fact, caused by the implementation details.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “[63] show that high performances of actiondependent baselines [19, 17, 37, 20, 67] were more directly due to different subtle implementation choices.”
Bootstrapped Policy Gradient for Difficulty Adaptation in Intelligent Tutoring Systems (2019, methodology): “Generally, the convergence of value function-based algorithms is not guaranteed [8, 26, 29].”
Understanding the impact of entropy on policy optimization (2018, background): “As such high variance in gradient estimates is often seen as a major issue for which several solutions have been proposed (Konda & Tsitsiklis, 2000; Greensmith et al., 2004; Schulman et al., 2015b; Tucker et al., 2018).”
Reinforcement learning for control: Performance, stability, and deep approximators (2018, methodology): “mance of a method than the parameters that are the focus of the work (Henderson et al., 2017; Tucker et al., 2018).”
What Matters for Adversarial Imitation Learning? (2021, background): “The performance of these high-level choices may also depend on low-level implementation details which might be silenced in the original publications [19, 29, 36, 42], as well as the hyperparameters (HPs) used.”