Deep Reinforcement Learning and the Deadly Triad

By Hado van Hasselt, Yotam Doron, Florian Strub, Matteo Hessel, Nicolas Sonnerat, Joseph Modayil. 2018

287 citations recorded by the source corpus.

We know from reinforcement learning theory that temporal difference learning can fail in certain cases. Sutton and Barto (2018) identify a deadly triad of function approximation, bootstrapping, and off-policy learning. When these three properties are combined, learning can diverge with the value estimates becoming unbounded. However, several algorithms successfully combine these three properties, which indicates that there is at least a partial gap in our understanding. In this work, we investigate the impact of the deadly triad in practice, in the context of a family of popular deep reinforcement learning models - deep Q-networks trained with experience replay - analysing how the components of this system play a role in the emergence of the deadly triad, and in the agent's performance

Subfields: Artificial Intelligence, Computer Networks and Communications, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, background): “The idea of using a slow-moving average target network to produce stable targets for the online network was inspired by deep RL [50, 51, 52, 53].”
Bayesian Structure Learning with Generative Flow Networks (2022, background): “Moreover, again inspired by Deep Q-learning (Van Hasselt et al., 2018), we found it advantageous to evaluate P ¯ θ ( s f | G (cid:48) ) in (10) with a separate target network—where the parameters ¯ θ are updated periodically.”
When to use parametric models in reinforcement learning? (2019, methodology): “However, just because divergence can occur does not mean it does occur [cf. van Hasselt et al., 2018].”
Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems (2020, background): “However, recent work suggests that high-capacity function approximators, which correspond to a very large set Q, generally do tend to make this method convergent in practice, yielding a Q-function that is close to ~Qπ (Fu et al., 2019; Van Hasselt et al., 2018).”
MOReL : Model-Based Offline Reinforcement Learning (2020, background): “itutes a major challenge in ofﬂine RL. This distribution shift is particularly exacerbated in MDPs with continuous state-action spaces owing to challenges in function approximation and generalization [14,15]. In principle, it is possible to use any off-policy RL algorithm (e.g. Q-learning, actor-cri”
Behavior Regularized Offline Reinforcement Learning (2019, methodology): “While oﬀ-policy (deep) RL algorithms such as DQN (Mnih et al., 2013), DDPG (Lillicrap et al., 2015), and SAC (Haarnoja et al., 2018) may be run directly on oﬄine datasets to learn a policy, the performance of these algorithms has been shown to be sensitive to the experience dataset distribution, eve”
Diagnosing Bottlenecks in Deep Q-learning Algorithms (2019, background): “Concurrently to us, Van Hasselt et al. (2018) experimentally ﬁnd that unbounded divergence rarely occurs with DQN variants on Atari games.”
Information-Theoretic Considerations in Batch Reinforcement Learning (2019, background): “The conjectured insufﬁciency of realizability (Conjecture 8) is re-lated to various undesirable phenomena in learning with bootstrapped targets, which has been of constant interest to RL researchers (Sutton, 2015; Van Hasselt et al., 2018; Lu et al., 2018).”
Towards Characterizing Divergence in Deep Q-Learning (2019, background): “Because divergence typically occurs near the beginning of training (van Hasselt et al., 2018) and the NTK is known to converge to a constant in the inﬁnite-width limit (Jacot et al., 2018), we focus only on the properties of K θ at initialization in these experiments.”
A survey and critique of multiagent deep reinforcement learning (2018, methodology): “In particular, function approximation, bootstrapping, and off-policy learning are considered the three main properties that when combined, can make the learning to diverge and are known as the deadly triad [315,334].”
Training Agents using Upside-Down Reinforcement Learning (2019, background): “This combination of techniques — referred to by Sutton and Barto [2018] as the deadly triad — can lead to instabilities which are usually addressed by adding a variety of ingredients, though complete remedies remain elusive [van Hasselt et al., 2018].”
Single and Multi-Agent Deep Reinforcement Learning for AI-Enabled Wireless Networks: A Tutorial (2020, background): “This is commonly referred to as the deadly triad: function approximation, bootstrapping, and off-policy training [51].”
Revisiting Fundamentals of Experience Replay (2020, methodology): “Function approximation of Q-values, bootstrapping, and off-policy learning have been identified as the deadly triad (Sutton & Barto, 2018; van Hasselt et al., 2018) of properties that, when combined, can negatively affect learning or even cause divergence. van Hasselt et al. (2018) suggest that…”
Shared Experience Actor-Critic for Multi-Agent Reinforcement Learning (2020, methodology): “However, we ﬁnd deep off-policy methods to exhibit rather unstable learning [14] compared to on-policy AC.”
The problem with DDPG: understanding failures in deterministic environments with sparse rewards (2019, background): “Meanwhile, van Hasselt et al. (2018) shows that, due to its stabilizing heuristics, DQN does not diverge much in practice when applied to the ATARI domain.”
Deep Reinforcement Learning for Task Offloading in Mobile Edge Computing Systems (2020, background): “, [29]), the convergence guarantee of a DRLalgorithm is still an open problem.”
Learning to Reach Goals via Iterated Supervised Learning (2019, background): “…especially those using temporal difference learning, are highly sensitive to hyperparameters [Henderson et al., 2018] and face challenges of stability and optimization [Tsitsiklis and Van Roy, 1997, van Hasselt et al., 2018, Kumar et al., 2019a], making such algorithms difﬁcult to use in practice.”
Is Conditional Generative Modeling all you need for Decision-Making? (2022, methodology): “These methods make use of function approximation, off-policy learning, and bootstrapping, leading to several instabilities in practice (Sutton, 1988; Van Hasselt et al., 2018).”