Deep Reinforcement Learning with Double Q-Learning
By Hado van Hasselt, Arthur Guez, David Silver. 2015
9,251 citations recorded by the source corpus.
The popular Q-learning algorithm is known to overestimate action values under certain conditions. It was not previously known whether, in practice, such overestimations are common, whether they harm performance, and whether they can generally be prevented. In this paper, we answer all these questions affirmatively. In particular, we first show that the recent DQN algorithm, which combines Q-learning with a deep neural network, suffers from substantial overestimations in some games in the Atari 2600 domain. We then show that the idea behind the Double Q-learning algorithm, which was introduced in a tabular setting, can be generalized to work with large-scale function approximation. We propose a specific adaptation to the DQN algorithm and show that the resulting algorithm not only reduces the observed overestimations, as hypothesized, but that this also leads to much better performance on several games.
Subfields: Artificial Intelligence, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
- Chelsea Finn
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Chelsea Finn
- Shimon Whiteson
- Ofir Nachum
- Yuxi Li
- Shixiang Shane Gu
- Marc G. Bellemare
Citations in the corpus
- Overcoming catastrophic forgetting in neural networks (2016, methodology): “We trained the networks with the Double Q-learning algorithm [van Hasselt et al., 2016].”
- Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “The asynchronous meth-ods tend to learn faster than DQN, with significantly faster learning on some games, while training on only 16 CPU cores.”
- Reward learning from human preferences and demonstrations in Atari (2018, methodology): “The algorithm we choose for reinforcement learning with expert demonstrations is deep Q-Learning from demonstrations (DQfD; Hester et al., 2018), which builds upon DQN (Mnih et al., 2015) and some of its extensions (Schaul et al., 2015; Wang et al., 2016; Hasselt et al., 2016).”
- Human-centric Dialog Training via Offline Reinforcement Learning (2020, background): “A second Target Q -network approximates the expected reward from the next state, Q θ T ( s t +1 , a t +1 ) (Van Hasselt et al., 2016).”
- Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learning (2019, background): “But these methods can be notoriously unstable, and in practice, require a variety of stabilization techniques to ensure more consistent performance (Hasselt et al., 2016; Wang et al., 2016; Munos et al., 2016; Hessel et al., 2017; Fujimoto et al., 2018; Nachum et al., 2018; Fu et al., 2019).”
- Addressing Function Approximation Error in Actor-Critic Methods (2018, methodology): “This section introduces a novel clipped variant of Double Q-learning (Van Hasselt, 2010), which can replace the critic in any actor-critic method.”
- Trust-PCL: An Off-Policy Trust Region Method for Continuous Control (2017, background): “Value-based RL algorithms such as Q-learning (Watkins, 1989) and policy-based algorithms such as actor-critic (Konda & Tsitsiklis, 2000) have achieved well known successes in environments with enumerable action spaces and predictable but possibly complex dynamics, e.g., as in Atari games (Mnih et al”
- The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, neutral): “Similar reasoning led to the introduction of Double DQN algo-rithm (Van Hasselt et al., 2016).”
- PGQ: Combining policy gradient and Q-learning (2016, methodology): “Recently, the field of deep reinforcement learning, i.e. , the use of deep neural networks to represent action-values or a policy, has seen a lot of success (Mnih et al., 2015; 2016; Silver et al., 2016; Riedmiller, 2005; Lillicrap et al., 2015; Van Hasselt et al., 2016).”
- Taming the Noise in Reinforcement Learning via Soft Updates (2015, methodology): “An interesting modification of Q-learning to address this problem is Double-Qlearning [11, 14], which uses two estimators for the Q function to alleviate the bias.”
- Off-Policy Actor-Critic with Shared Experience Replay (2019, methodology): “For evaluation, we average episode returns within buckets of 1M (Atari) and 10M (DMLab) environment steps for each agent instance, and normalize scores on each game by using the scores of a human expert and a random agent (van Hasselt et al., 2016).”
- When to use parametric models in reinforcement learning? (2019, methodology): “…to Rainbow DQN [Hessel et al., 2018a], which combines the DQN algorithm [Mnih et al., 2013, 2015] with double Q-learning [van Hasselt, 2010, van Hasselt et al., 2016], dueling network architectures [Wang et al., 2016], prioritised experience replay [Schaul et al., 2016], noisy networks for…”
- Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “Specifically, GATS deploys the MCTS method for planning over a bounded tree depth and uses the DQN algorithm to estimate the Q -function as a value for the leaf states (Mnih et al., 2015; Van Hasselt et al., 2016).”
- Observe and Look Further: Achieving Consistent Performance on Atari (2018, background): “Let t 1 , ..., t N be N transitions from the buffer with normalized priorities p 1 , ..., p N , then for k ∈ N the loss function in (1) using the operator T h is approximated as ) where a i := argmax a ∈A f θ ( k − 1) ( x i , a ) for DQN and a i := argmax a ∈A f θ ( x i , a ) for Double DQN [20].”
- Distributed Prioritized Experience Replay (2018, methodology): “More specifically, we used double Q-learning (van Hasselt, 2010; van Hasselt et al., 2016) with multi-step bootstrap targets (cf. Sutton, 1988; Sutton & Barto, 1998; 2017; Mnih et al., 2016) as the learning algorithm, and a dueling network architecture (Wang et al., 2016) as the function…”
- Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “Pop-Art normalization (van Hasselt et al. 2016) allows reward clipping to be removed, while preserving a similar level of performance.”
- Prioritized Experience Replay (2015, methodology): “Our principal modification is to replace the uniform random sampling used by Double DQN with our stochastic prioritization and importance sampling methods (see Algorithm 1).”