ConQUR: Mitigating Delusional Bias in Deep Q-learning
By A. Su, Jayden Ooi, Tyler Lu, Dale Schuurmans, Craig Boutilier. 2020
5 citations recorded by the source corpus.
Delusional bias is a fundamental source of error in approximate Q-learning. To date, the only techniques that explicitly address delusion require comprehensive search using tabular value estimates. In this paper, we develop efficient methods to mitigate delusional bias by training Q-approximators with labels that are "consistent" with the underlying greedy policy class. We introduce a simple penalization scheme that encourages Q-labels used across training batches to remain (jointly) consistent with the expressible policy class. We also propose a search framework that allows multiple Q-approximators to be generated and tracked, thus mitigating the effect of premature (implicit) policy commitments. Experimental results demonstrate that these methods can improve the performance of Q-learning in a variety of Atari games, sometimes dramatically.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Yoshua Bengio
- Sergey Levine
- Caglar Gulcehre
- Kyunghyun Cho
- Quoc V. Le
- Ruslan Salakhutdinov
- Ziyun Wang
- Chelsea Finn
Citations in the corpus
- Regularized Behavior Value Estimation (2021, methodology): “Previously, hinge losses were used for regularization in RL, but with different goals from ours (Su et al., 2020; Pohlen et al., 2018).”