Maxmin Q-learning: Controlling the Estimation Bias of Q-learning

By Qingfeng Lan, Yangchen Pan, Alona Fyshe, Martha White. 2020

219 citations recorded by the source corpus.

Q-learning suffers from overestimation bias, because it approximates the maximum action value using the maximum estimated action value. Algorithms have been proposed to reduce overestimation bias, but we lack an understanding of how bias interacts with performance, and the extent to which existing algorithms mitigate bias. In this paper, we 1) highlight that the effect of overestimation bias on learning efficiency is environment-dependent; 2) propose a generalization of Q-learning, called \emph{Maxmin Q-learning}, which provides a parameter to flexibly control bias; 3) show theoretically that there exists a parameter choice for Maxmin Q-learning that leads to unbiased estimation with a lower approximation variance than Q-learning; and 4) prove the convergence of our algorithm in the tabular case, as well as convergence of several previous Q-learning variants, using a novel Generalized Q-learning framework. We empirically verify that our algorithm better controls estimation bias in toy environments, and that it achieves superior performance on several benchmark problems.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “These approximators constitute naturally an ensemble, the size of which controls the intensity of underestimation: more approximators correspond to more severe underestimation (Lan et al., 2020).”
Memory-efficient Reinforcement Learning with Knowledge Consolidation (2022, methodology): “For DQN, we used the same hyper-parameters and training settings as in Lan et al. (2020).”
Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble (2021, background): “Subsequently, Maxmin Q-learning [17] theoretically shows that the overestimation bias can be controlled by the number of ensembles in the clipped Q-learning.”
Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, methodology): “Following Thrun & Schwartz (1993) and Lan et al. (2020), ﬁx s and assume each Q i ( s, a ) has a random approximation error e : where for each ﬁxed s , { e } are zero-mean independent random variables such that { e } are identically distributed across i for each ﬁxed ( s, a ) pair.”