Variance Reduction for Policy Gradient with Action-Dependent Factorized Baselines

By Cathy Wu, Aravind Rajeswaran, Yan Duan, Vikash Kumar, Alexandre M. Bayen, Sham M. Kakade, Igor Mordatch, Pieter Abbeel. 2018

173 citations recorded by the source corpus.

Policy gradient methods have enjoyed great success in deep reinforcement learning but suffer from high variance of gradient estimates. The high variance problem is particularly exasperated in problems with long horizons or high-dimensional action spaces. To mitigate this issue, we derive a bias-free action-dependent baseline for variance reduction which fully exploits the structural form of the stochastic policy itself and does not make any additional assumptions about the MDP. We demonstrate and quantify the benefit of the action-dependent baseline through both theoretical analysis as well as numerical results, including an analysis of the suboptimality of the optimal state-dependent baseline. The result is a computationally efficient policy gradient algorithm, which scales to high-dimensional control problems, as demonstrated by a synthetic 2000-dimensional target matching task. Our experimental results indicate that action-dependent baselines allow for faster learning on standard reinforcement learning benchmarks and high-dimensional hand manipulation and synthetic tasks. Finally, we show that the general idea of including additional information in baselines for improved variance reduction can be extended to partially observed and multi-agent tasks.

Subfields: Artificial Intelligence, Computational Theory and Mathematics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computational Theory and Mathematics

In Reinforcement Learning

Citations in the corpus

The Mirage of Action-Dependent Baselines in Reinforcement Learning (2018, methodology): “Several recent methods (Gu et al., 2017a; Thomas & Brunskill, 2017; Grathwohl et al., 2018; Liu et al., 2018; Wu et al., 2018) have extended the approach to state-action-dependent baselines (i.”
From Importance Sampling to Doubly Robust Policy Gradient (2019, neutral): “(2019) noticed that the previous variance reduction methods in PG overlooked the correlation across the times steps and ignored the randomness in the future steps (e.g., Grathwohl et al., 2018; Gu et al., 2017; Liu et al., 2018; Wu et al., 2018).”
Trajectory-wise Control Variates for Variance Reduction in Policy Gradient Methods (2019, background): “In practice, it has been observed that these CVs indeed accelerate policy optimization, especially in simulated robot control tasks [20, 21, 23, 25, 36].”
Reward-estimation variance elimination in sequential decision processes (2018, background): “More recently state-action-dependent baselines have been investigated [2, 4, 8, 20].”
Backpropagation through the Void: Optimizing control variates for black-box gradient estimation (2017, methodology): “ al.(2016) and Gu et al.(2017) which train an action-dependent baseline which incorporates off-policy data.Liu et al.(2017) independently develop a method similar to LAX applied to continuous control.Wu et al. (2018) exploit per-dimension independence of the action distribution in continuous control”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, background): “Policy gradients, however, can suffer from large variance that may limit performance, especially for high-dimensional action spaces (Wu et al., 2018).”
CM3: Cooperative Multi-goal Multi-stage Multi-agent Reinforcement Learning (2018, background): “[44] suggests that (1) is a low-variance estimator.”
Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, background): “[63] show that high performances of actiondependent baselines [19, 17, 37, 20, 67] were more directly due to different subtle implementation choices.”
A Game Theoretic Framework for Model Based Reinforcement Learning (2020, methodology): “Finally, we use a baseline/value network for the purposes of variance reduction (Greensmith et al., 2001; Wu et al., 2018) – specifically GAE (Schulman et al., 2016).”
Actor-Critic Policy Optimization in Partially Observable Multiagent Environments (2018, methodology): “an advantage actor-critic algorithm differing from A2C in the (state-action) representation of the critics [56, 96] and summing over actions similar to the all-action algorithms [86, 71, 19, 5].”
Action-depedent Control Variates for Policy Optimization via Stein's Identity (2017, background): “Wu et al. (2018) exploit per-dimension independence of the action distribution to produce an action-dependent baseline in continuous control tasks.”
Deep Reinforcement Learning for Cyber Security (2019, background): “The gradient estimation, however, has often suffered a large fluctuation [62].”
Counterfactual Credit Assignment in Model-Free Reinforcement Learning (2020, background): “…when using independent multi-dimensional actions, the baseline for one dimension of the action vector can include the actions in other dimensions (Wu et al., 2018); or when the dynamic of the environment is partially driven by an exogenous and stochastic factor, independent of the agent’s…”
Statistically Efficient Off-Policy Policy Gradients (2020, background): “Variance reduction has been a central topic for PG (Greensmith et al., 2004; Schulman et al., 2016; Tang and Abbeel, 2010; Wu et al., 2018).”
Policy Optimization via Importance Sampling (2018, background): “Baselines are typically constant (REINFORCE), time-dependent (G(PO)MDP) or state-dependent (actor-critic), but these approaches have been recently extended to account for action-dependent baselines [57, 64].”
Multi-agent deep reinforcement learning: a survey (2021, methodology): “Therefore, Wu et al. (2018) proposed an action-dependent baseline which includes information from other agents to reduce the variance in the critic estimation function.”
Emergence of Scenario-Appropriate Collaborative Behaviors for Teams of Robotic Bodyguards (2018, unknown): “Several different methods such as (Wu et al. 2018; Schulman et al. 2017) have been shown to reduce the variability in policy gradient methods by introducing a critic which is a Q-function that tells about the goodness of a reward by working as a baseline.”