Bias-corrected Q-learning to control max-operator bias in Q-learning

By Donghun Lee, Boris Defourny, Warrren B Powell. 2013

53 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Ofir Nachum
4. Marc G. Bellemare
5. Scott Fujimoto
6. Shixiang Shane Gu
7. Shimon Whiteson
8. R´emi Munos

In Management Science and Operations Research

1. Marc G. Bellemare
2. Sergey Levine
3. Scott Fujimoto
4. Herke Van Hoof
5. R´emi Munos
6. Georg Ostrovski
7. David Meger
8. Marcello Restelli

In Reinforcement Learning

1. Sergey Levine
2. Pieter Abbeel
3. Ofir Nachum
4. Marc G. Bellemare
5. Scott Fujimoto
6. Shixiang Shane Gu
7. Aviral Kumar
8. George Tucker

Citations in the corpus

• Addressing Function Approximation Error in Actor-Critic Methods (2018, background): “Other approaches have focused directly on reducing the variance (Anschel et al., 2017), minimizing over-fitting to early high variance estimates (Fox et al., 2016), or through corrective terms (Lee et al., 2013).”
• Increasing the Action Gap: New Operators for Reinforcement Learning (2015, background): “…a similar effect on the value estimates when replacing the Bell-man updates with Double Q-Learning updates, one of many solutions recently proposed to mitigate the negative impact of statistical bias in value function estimation (van Hasselt 2010; Azar et al. 2011; Lee, Defourny, and Powell 2013).”
• Interleaved Q-Learning with Partially Coupled Training Process (2019, unknown): “phased Q-learning [10], fitted Q-iteration [7], bias-corrected Qlearning [12] and deep Q-networks (DQN) [13].”
• Maxmin Q-learning: Controlling the Estimation Bias of Q-learning (2020, unknown): “Bias-corrected Q-Learning (Lee et al., 2013) reduces the overestimation bias through a bias correction term.”
• Estimating the Maximum Expected Value in Continuous Reinforcement Learning Problems (2017, background): “Under the assumption of Gaussian rewards, in (Lee, Defourny, and Powell 2013) the authors propose a Q-learning variant that corrects the positive bias of ME by subtracting a term that depends on the number of actions and the variance of the rewards.”
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, methodology): “Other bias reduction techniques include using bias-correction terms (Lee et al., 2013), using weighted Q estimates (Zhang et al., 2017; Li & Hou, 2019), penalizing deterministic policies at early stage of training (Fox et al., 2015), using multi-step methods (Meng et al., 2020), performing weighted…”