Addressing Function Approximation Error in Actor-Critic Methods
By Scott Fujimoto, Herke Van Hoof, David Meger. 2018
7,119 citations recorded by the source corpus.
In value-based reinforcement learning methods such as deep Q-learning, function approximation errors are known to lead to overestimated value estimates and suboptimal policies. We show that this problem persists in an actor-critic setting and propose novel mechanisms to minimize its effects on both the actor and the critic. Our algorithm builds on Double Q-learning, by taking the minimum value between a pair of critics to limit overestimation. We draw the connection between target networks and overestimation bias, and suggest delaying policy updates to reduce per-update error and further improve performance. We evaluate our method on the suite of OpenAI gym tasks, outperforming the state of the art in every environment tested.
Subfields: Artificial Intelligence, Reinforcement Learning, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Ruslan Salakhutdinov
- Mohit Bansal
- Quoc V. Le
- Trevor Darrell
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Ofir Nachum
- Shixiang Shane Gu
- Aviral Kumar
- George Tucker
- Shimon Whiteson
In Optimization & Training
- Sergey Levine
- Yoshua Bengio
- Quoc V. Le
- Chelsea Finn
- Pieter Abbeel
- Razvan Pascanu
- Kyunghyun Cho
- Frank Hutter
Citations in the corpus
- Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialog (2019, methodology): “Since BCQ relies on Double Clipped Q-learning [15], here we use dropout-based uncertainty estimates as in Eq.”
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “DDPG, TD3, and SAC are not easily applicable to discrete action spaces due to their need to back-propagate from the Q-function to the policy.”
- Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor (2018, methodology): “We then use the minimum of the Q-functions for the value gradient in Equation 6 and policy gradient in Equation 13, as proposed by Fujimoto et al. (2018).”
- Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “We also estimate these values for baselines, including the minimum predicted Q-value under an ensemble [19, 16] of Qfunctions with varying ensemble sizes, which is a standard technique to prevent overestimed Q-values [16, 19, 21] and BEAR [30], a policy constraint method.”
- Accelerating Online Reinforcement Learning with Offline Datasets (2020, methodology): “We utilize a standard temporal difference learning scheme for policy evaluation [7, 4], by minimizing the Bellman error.”
- CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learning (2020, methodology): “2), we train agents under the three described curricula using the following MF-RL algorithms: The original Proximal Policy Optimization (PPO) from Schulman et al. (2017), Soft Actor-Critic (SAC) from Haarnoja et al. (2018) and the Twin Delayed DDPG (TD3) from Fujimoto et al. (2018).”
- Off-Policy Deep Reinforcement Learning without Exploration (2018, methodology): “Clipped Double Q-learning (Fujimoto et al., 2018), which estimates the value by taking the minimum between two Qnetworks {Qθ1 , Qθ2}.”
- Adaptive Online Planning for Continual Lifelong Learning (2019, methodology): “We present a new algorithm, Adaptive Online Planning (AOP), that links Model Predictive Path Integral control (MPPI) [32], a model-based planner, with Twin Delayed DDPG (TD3) [12], a model-free policy learning method.”
- MOPO: Model-based Offline Policy Optimization (2020, background): “While off-policy RL algorithms [42, 26, 19] can in principle utilize previously collected datasets, they perform poorly without online data collection.”
- MOReL : Model-Based Offline Reinforcement Learning (2020, background): “In principle, any off-policy algorithm based on Qlearning [50, 51] or actor-critic architectures [52, 14, 53] can be used with a static offline dataset.”
- DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “For continuous control domains, we use the clipped double Q-learning trick (Fujimoto et al., 2018), which is also referred to as the “twin-Q” trick, and it further parametrizes another pair of online and target Q-functions, and uses the minimum Q-value for backup computation.”
- GradientDICE: Rethinking Generalized Offline Estimation of Stationary Values (2020, methodology): “Our target policy π is a policy trained via TD3 (Fujimoto et al., 2018) for 10 6 steps.”
- AlgaeDICE: Policy Gradient from Arbitrary Experience (2019, methodology): “We follow the evaluation protocol of Fujimoto et al. (2018), plotting the performance of 10 randomly seeded training runs, with shaded region representing half a standard deviation and x -axis given by environment steps.”
- Behavior Regularized Offline Reinforcement Learning (2019, methodology): “…have been proposed recently: 1) Agarwal et al. (2019) proposes to apply a random ensemble of Q-value targets to stabilize the learned Q-function, 2) Fujimoto et al. (2018a); Kumar et al. (2019); Jaques et al. (2019); Laroche & Trichelair (2017) propose to regularize the learned policy towards the…”
- Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, methodology): “Actor-critic methods [36, 14, 19] address this by additionally learning a policy π θ that maximizes the Q -function.”
- Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, methodology): “We further evaluate AFM on MuJoCo tasks with the TD3 al-gorithm (Fujimoto et al., 2018) and the entropy constrained SAC algorithm (Haarnoja et al., 2018).”
- Learning Self-Correctable Policies and Value Functions from Demonstrations with Negative Sampling (2019, methodology): “We develop an algorithm that learns a value function that extrapolates to unseen states more conservatively, as an approach to attack the optimistic extrapolation problem [Fujimoto et al., 2018a].”