Dueling Network Architectures for Deep Reinforcement Learning
By Ziyun Wang, Tom Schaul, Matteo Hessel, Hado van Hasselt, Marc Lanctot, Nando de Freitas. 2015
4,476 citations recorded by the source corpus.
In recent years there have been many successes of using deep representations in reinforcement learning. Still, many of these applications use conventional architectures, such as convolutional networks, LSTMs, or auto-encoders. In this paper, we present a new neural network architecture for model-free reinforcement learning. Our dueling network represents two separate estimators: one for the state value function and one for the state-dependent action advantage function. The main benefit of this factoring is to generalize learning across actions without imposing any change to the underlying reinforcement learning algorithm. Our results show that this architecture leads to better policy evaluation in the presence of many similar-valued actions. Moreover, the dueling architecture enables our RL agent to outperform the state-of-the-art on the Atari 2600 domain.
Subfields: Artificial Intelligence, Automotive Engineering, Reinforcement Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Yoshua Bengio
- Pieter Abbeel
- Mohit Bansal
- Ruslan Salakhutdinov
- Quoc V. Le
- Jianfeng Gao
- Percy Liang
In Automotive Engineering
- Sergey Levine
- Pieter Abbeel
- Lei Tai
- R´emi Munos
- Chelsea Finn
- Ofir Nachum
- Marc G. Bellemare
- David Silver
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Shimon Whiteson
- Yuxi Li
- Chelsea Finn
- Ofir Nachum
- Marc G. Bellemare
- Shixiang Shane Gu
Citations in the corpus
- What can the brain teach us about building artificial intelligence? (2016, background): “, 2016) and 96% by using smarter replay and more efficient parameter sharing (Wang et al., 2016) (see DQN+ and DQN++ in Fig.”
- Sample Efficient Actor-Critic with Experience Replay (2016, methodology): “Experience replay (Lin, 1992) has gained popularity in deep Q-learning (Mnih et al., 2015; Schaul et al., 2016; Wang et al., 2016; Narasimhan et al., 2015), where it is often motivated as a technique for reducing sample correlation.”
- Asynchronous Methods for Deep Reinforcement Learning (2016, methodology): “We note that many of the improvements that are presented in Double DQN (Van Hasselt et al., 2015) and Dueling Double DQN (Wang et al., 2015) can be incorporated to 1-step Q and n-step Q methods presented in this work with similar potential improvements.”
- Reward learning from human preferences and demonstrations in Atari (2018, methodology): “The algorithm we choose for reinforcement learning with expert demonstrations is deep Q-Learning from demonstrations (DQfD; Hester et al., 2018), which builds upon DQN (Mnih et al., 2015) and some of its extensions (Schaul et al., 2015; Wang et al., 2016; Hasselt et al., 2016).”
- Choose Your Weapon: Survival Strategies for Depressed AI Academics (2023, background): “The successful examples are out there [21, 22, 3], but more is needed from the University’s end to enable such partnerships.”
- Equivalence Between Policy Gradients and Soft Q-Learning (2017, neutral): “The Q-function was represented as: Qθ(s, a) = Vθ(s) + τ log πθ(a | s), which can be seen as a form of dueling architecture with τ log πθ(a | s) being the “advantage stream” (Wang et al. [2015]).”
- The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, methodology): “As described in Section 2.1, the Retrace algorithm makes use of two policies, the behaviour and target policies.”
- PGQ: Combining policy gradient and Q-learning (2016, methodology): “If η = 1 then it becomes a variant of (batch) Q-learning with an architecture similar to the dueling architecture (Wang et al., 2016).”
- When to use parametric models in reinforcement learning? (2019, methodology): “…the DQN algorithm [Mnih et al., 2013, 2015] with double Q-learning [van Hasselt, 2010, van Hasselt et al., 2016], dueling network architectures [Wang et al., 2016], prioritised experience replay [Schaul et al., 2016], noisy networks for exploration [Fortunato et al., 2017], and distributional…”
- Observe and Look Further: Achieving Consistent Performance on Atari (2018, methodology): “We first analyse the performance of the standard dueling DQN architecture [22] that is also used by the baselines.”
- Distributed Prioritized Experience Replay (2018, methodology): “As a result, prioritized experience replay is used in many agents, such as Prioritized Dueling DQN (Wang et al., 2016), UNREAL (Jaderberg et al.”
- Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, background): “It features two streams of computation, the value and advantage streams, sharing a convolutional encoder, and merged by a special aggregator (Wang et al. 2016).”
- Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, background): “Note also that the best score achieved across training is a statistically biased estimate of an agent’s best performance: to avoid this bias, one should perform a second, independent evaluation of the agent at that particular point in time, as reported by Wang et al. (2016).”
- Value Prediction Network (2017, background): “Mnih et al. [21] proposed the Deep Q-Network (DQN) architecture which learns to estimate Q-values using deep neural networks.”
- Reinforcement Learning with Unsupervised Auxiliary Tasks (2016, methodology): “A3C and Prioritized Dueling DQN (Wang et al., 2016), across all levels attaining 880% mean and 250% median performance.”
- Prioritized Experience Replay (2015, neutral): “Simultaneously with our work, an architectural innovation that separates advantages from the value function (see the co-submission by Wang et al., 2015) has also led to substantial improvements on the Atari benchmark.”
- Neural Approaches to Conversational AI (2018, positive): “There have been a number of recent improvements to the basic Q-learning described above, such as dueling Q-network (Wang et al., 2016), double Q-learning (van Hasselt et al.”