Diagnosing Bottlenecks in Deep Q-learning Algorithms

By Justin Fu, Aviral Kumar, Matthew Soh, Sergey Levine. 2019

153 citations recorded by the source corpus.

Q-learning methods represent a commonly used class of algorithms in reinforcement learning: they are generally efficient and simple, and can be combined readily with function approximators for deep reinforcement learning (RL). However, the behavior of Q-learning methods with function approximation is poorly understood, both theoretically and empirically. In this work, we aim to experimentally investigate potential issues in Q-learning, by means of a "unit testing" framework where we can utilize oracles to disentangle sources of error. Specifically, we investigate questions related to function approximation, sampling error and nonstationarity, and where available, verify if trends found in oracle settings hold true with modern deep RL methods. We find that large neural network architectures have many benefits with regards to learning stability; offer several practical compensations for overfitting; and develop a novel sampling method based on explicitly compensating for function approximation error that yields fair improvement on high-dimensional continuous control domains.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, methodology): “Furthermore, it can be difﬁcult to apply these methods to learn from fully off-policy data, where an agent is unable to collect additional environmental interactions (Fujimoto et al., 2019; Kumar et al., 2019).”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “For example, when the size of the dataset is limited, approximate dynamic programming algorithms tend to overfit on the small dataset, and this error is then accumulated via Bellman backups (Fu et al., 2019; Kumar et al., 2020).”
Conservative Q-Learning for Offline Reinforcement Learning (2020, methodology): “Directly utilizing existing value-based off-policy RL algorithms in an offline setting generally results in poor performance, due to issues with bootstrapping from out-of-distribution actions [30, 15] and overfitting [13, 30, 3].”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “…o rr e la ti o n Ek+1 − Ek Bellman error To analyze this phenomenon computationally, we use the gridworld MDPs from Fu et al. (2019) and visualize the correlations between policy visitations dπk(s, a) and the value of Bellman error after the ADP up- date, i.e.…”
Behavior Regularized Offline Reinforcement Learning (2019, methodology): “We brieﬂy review two recently proposed approaches, BEAR (Kumar et al., 2019) and BCQ (Fujimoto et al., 2018a).”
Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction (2019, background): “This issue persists even when the off-policy data comes from effective expert policies, which in principle should address any exploration challenge [7, 13, 12].”
Towards Characterizing Divergence in Deep Q-Learning (2019, background): “Also concurrently, Fu et al. (2019) performed a rigorous empirical study of Fit-ted Q-Iteration (FQI) (Riedmiller, 2005) to gain insight into divergence issues in DQL, and ultimately proposed an algo-rithm based on data distribution modiﬁcations to improve performance.”
Reward-Conditioned Policies (2019, background): “…tasks [13, 20, 22, 28, 36], they are also known to be notoriously challenging to use effectively, due to sensitivity to hyperparameters, high sample complexity, and a range of important and delicate implementation choices that have a large effect on performance [5, 6, 12, 15, 23, 24, 46].”
You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL (2021, unknown): “Specifically, actor-critic based off-policy RL algorithms, which iterate policy evaluation and improvement, suffer from the overestimation problem caused by function approximation error and bootstrapping [2, 3] in the offline RL setup, even though the algorithms are equipped with algorithmic techniq”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, background): “Similar to our work, empirical properties of deep value functions which induce instability or divergence have been studied (Fu et al., 2019; Achiam et al., 2019) but have not considered the role of the objective itself.”
COG: Connecting New Skills to Past Experience with Offline Reinforcement Learning (2020, background): “These works largely focuses on developing more effective ofﬂine deep RL algorithms by correcting for distribution shift [41, 3, 4, 38, 39, 40] which renders standard off-policy RL algorithms inadmissible in purely ofﬂine settings [4, 1].”
Towards Understanding Linear Value Decomposition in Cooperative Multi-Agent Q-Learning (2020, methodology): “From an empirical perspective, there is also a growing trend to adopt FQI for empirical analysis of deep offline Q-learning algorithms (Fu et al., 2019; Levine et al., 2020).”
Revisiting Fundamentals of Experience Replay (2020, background): “Fu et al. (2019) report that agent performance is sensitive to the number of environment steps taken per gradient step, with too small or large a ratio also hindering performance. van Hasselt et al. (2019) vary this ratio in combination with batch sizes to obtain a more sample-efficient version of…”
On Value Functions and the Agent-Environment Boundary (2019, methodology): “, 2015], and often used as a representative of off-policy value-based RL algorithms in empirical studies [Fu et al., 2019].”
Regret Minimization Experience Replay in Off-Policy Reinforcement Learning (2021, background): “Adversarial Feature Matching [18] focuses on sampling uniformly among state-action pairs.”
DDPG++: Striving for Simplicity in Continuous-control Off-Policy Reinforcement Learning (2020, background): “, [23] is a thorough study of Q-learning algorithms, [4] identifies the key concept of overestimation bias in Q-learning, [24] shows that offline Q-learning, i.”