Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learning

By Oron Anschel, Nir Baram, Nahum Shimkin. 2016

367 citations recorded by the source corpus.

Instability and variability of Deep Reinforcement Learning (DRL) algorithms tend to adversely affect their performance. Averaged-DQN is a simple extension to the DQN algorithm, based on averaging previously learned Q-values estimates, which leads to a more stable training procedure and improved performance by reducing approximation error variance in the target values. To understand the effect of the algorithm, we examine the source of value function estimation errors and provide an analytical comparison within a simplified model. We further present experiments on the Arcade Learning Environment benchmark that demonstrate significantly improved stability and performance due to the proposed extension.

Subfields: Artificial Intelligence, Electrical and Electronic Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Electrical and Electronic Engineering

In Reinforcement Learning

Citations in the corpus

Addressing Function Approximation Error in Actor-Critic Methods (2018, background): “Other approaches have focused directly on reducing the variance (Anschel et al., 2017), minimizing over-ﬁtting to early high variance estimates (Fox et al., 2016), or through corrective terms (Lee et al., 2013).”
Fitted Q-Learning for Relational Domains (2020, neutral): “…(DQN), ﬁrst introduced by Mnih et al. (2015) and later extended by others (Hausknecht and Stone 2015; Van Hasselt, Guez, and Silver 2016; Mnih et al. 2016; Anschel, Baram, and Shimkin 2017) can also be seen as ﬁtted Q-iteration where the Q-functions are approximated by Deep Neural networks.”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, methodology): “Note that our Ensemble-DQN is different from the ensembling proposed in [1] which uses the average target Q-values for training the individual Q-functions.”
Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “Embodiments of this approach are based on dropout Anschel et al. (2017), employing previous Q-function approximations (Ghavamzadeh et al., 2011; An-schel et al., 2017), the linear combination between min and max over the pool of Q-networks (Li & Hou, 2019; Kumar et al., 2019), or the random mixture…”
Applications of reinforcement learning in energy systems (2021, background): “Thus, in recent years, the RL research community focuses on developing stable RL algorithms with reduced variance [169,170].”
Parametrized Deep Q-Networks Learning: Reinforcement Learning with Discrete-Continuous Hybrid Action Space (2018, background): “…of extensions are proposed to improve over DQN, including Double DQN [Hasselt et al., 2016], dueling DQN [Wang et al., 2016], bootstrap DQN [Osband et al., 2016], asynchronous DQN [Mnih et al., 2016], averaged-DQN Anschel et al. [2017] and prioritized experience replay [Schaul et al., 2016].”
An Optimistic Perspective on Offline Reinforcement Learning (2019, methodology): “Ensemble-DQN is a simple extension of DQN that approximates the Q-values via an ensemble of parameterized Qfunctions (Faußer & Schwenker, 2015; Osband et al., 2016; Anschel et al., 2017).”
ConQUR: Mitigating Delusional Bias in Deep Q-learning (2020, methodology): “One direct approach uses ensembling, with multiple Q-approximators updated in parallel (Faußer & Schwenker, 2015; Osband et al., 2016; Anschel et al., 2017) and combined to reduce instability and variance.”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, methodology): “Although average bias has been analyzed in several previous works (Van Hasselt et al., 2016; Fujimoto et al., 2018; Anschel et al., 2017), the std does not seem to have received much attention.”
Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning (2021, methodology): “To verify the generalization of UWAC to uncertainty estimation methods beyond dropout, we applied UWAC loss to ensembles trained under REM (Agarwal et al., 2020) and average-DQN ensembles (Anschel et al., 2017).”
Maxmin Q-learning: Controlling the Estimation Bias of Q-learning (2020, background): “For Averaged Q-learning, the proof is similar to Ensemble Q-learning except that N = 1 and K is a positive integer.”
Interference and Generalization in Temporal Difference Learning (2020, result): “These results are also consistent with Anschel et al. (2017) and Agarwal et al. (2019) who obtain improvements by training ensembles of value functions.”
Deep Reinforcement Learning: An Overview (2017, background): “, 2017a) reduce variability and instability with averaged-DQN (Anschel et al., 2017) – dueling architecture (Wang et al.”
The Reactor: A fast and sample-efficient Actor-Critic agent for Reinforcement Learning (2017, background): “act -greedily according to an action-value function and train using minibatches of transitions sampled from an experience replay buffer (Van Hasselt et al., 2016; Wang et al., 2015; He et al., 2017; Anschel et al., 2017). These value-function agents beneﬁt from improved sample complexity, but tend t”
Stochastic Double Deep Q-Network (2019, methodology): “[33] proposed averaged DQN (ADQN) based on the average estimator operation.”