Stochastic Double Deep Q-Network

By Pingli Lv, Xuesong Wang, Yuhu Cheng, Ziming Duan. 2019

29 citations recorded by the source corpus.

Estimation bias seriously affects the performance of reinforcement learning algorithms. The maximum operation may result in overestimation, while the double estimator operation often leads to underestimation. To eliminate the estimation bias, these two operations are combined together in our proposed algorithm named stochastic double deep Q-learning network (SDDQN), which is based on the idea of random selection. A tabular version of SDDQN is also given, named stochastic double Q-learning (SDQ). Both the SDDQN and SDQ are based on the double estimator framework. At each step, we choose to use either the maximum operation or the double estimator operation with a certain probability, which is determined by a random selection parameter. The theoretical analysis shows that there indeed exists a proper random selection parameter that makes SDDQN and SDQ unbiased. The experiments on Grid World and Atari 2600 games illustrate that our proposed algorithms can balance the estimation bias effectively and improve performance.

Subfields: Artificial Intelligence, Computational Theory and Mathematics

Citations in the corpus

• Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, neutral): “Subse-quently, (Zhang et al., 2017) and Lv et al. (2019) introduced the Weighted Estimator in the reinforcement learning community.”