Towards Characterizing Divergence in Deep Q-Learning

By Joshua Achiam, Ethan Knight, Pieter Abbeel. 2019

105 citations recorded by the source corpus.

Deep Q-Learning (DQL), a family of temporal difference algorithms for control, employs three techniques collectively known as the `deadly triad' in reinforcement learning: bootstrapping, off-policy learning, and function approximation. Prior work has demonstrated that together these can lead to divergence in Q-learning algorithms, but the conditions under which divergence occurs are not well-understood. In this note, we give a simple analysis based on a linear approximation to the Q-value updates, which we believe provides insight into divergence under the deadly triad. The central point in our analysis is to consider when the leading order approximation to the deep-Q update is or is not a contraction in the sup norm. Based on this analysis, we develop an algorithm which permits stable deep Q-learning for continuous control without any of the tricks conventionally used (such as target networks, adaptive gradient optimizers, or using multiple Q functions). We demonstrate that our algorithm performs above or near state-of-the-art on standard MuJoCo benchmarks from the OpenAI Gym.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Conservative Q-Learning for Offline Reinforcement Learning (2020, background): “As prior work has argued and shown [1, 12, 28], the “generalization” or the coupling effects of the function approximator may be heavily influenced by the properties of the data distribution [12, 28].”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, methodology): “While a number of prior works (Achiam et al., 2019; Fu et al., 2019; Liu and Zou, 2018) have provided a theoretical examination of various approximate dynamic programming (ADP) methods, prior work has not extensively studied the relationship between the data distribution induced by the latest value…”
Regularized Behavior Value Estimation (2021, methodology): “See Appendix A for a formal statement and proof on this statement, though similar observations have been made before by Fujimoto et al. (2019b) and Achiam et al. (2019).”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “In practice, using function approximators in RL requires making crucial representational decisions and poor design choices can result in estimates that diverge from the optimal value function [1,21,46,112,334,351].”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, background): “Similar to our work, empirical properties of deep value functions which induce instability or divergence have been studied (Fu et al., 2019; Achiam et al., 2019) but have not considered the role of the objective itself.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “Given what prior work has shown about RL in the presence of function approximation and state aliasing [2, 134, 138], it is not surprising that end-to-end learning of representations is fragile [72, 136]: an algorithm needs good representations to drive the learning of the RL algorithm, but the RL al”
Interference and Generalization in Temporal Difference Learning (2020, methodology): “In TD methods, models also face a different optimization procedure where different components may be at odds with each other, leading to phenomena like the deadly triad (Sutton & Barto, 2018; Achiam et al., 2019).”
A Fine-Grained Spectral Perspective on Neural Networks (2019, methodology): “Φ̃(x) =  Φ(x) if x ∈ [−1, 1] Φ(2− x) if x ∈ [1, 2] Φ(−2− x) if x ∈ [−2,−1].”