Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics

By Arsenii Kuznetsov, Pavel Shvechikov, Alexander Grishin, Dmitry P. Vetrov. 2020

282 citations recorded by the source corpus.

The overestimation bias is one of the major impediments to accurate off-policy learning. This paper investigates a novel way to alleviate the overestimation bias in a continuous control setting. Our method---Truncated Quantile Critics, TQC,---blends three ideas: distributional representation of a critic, truncation of critics prediction, and ensembling of multiple critics. Distributional representation and truncation allow for arbitrary granular overestimation control, while ensembling provides additional score improvements. TQC outperforms the current state of the art on all environments from the continuous control benchmark suite, demonstrating 25% improvement on the most challenging Humanoid environment.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Pieter Abbeel
3. Chelsea Finn
4. Shixiang Shane Gu
5. Ilya Kostrikov
6. Yoshua Bengio
7. Pratik Chaudhari
8. Ruslan Salakhutdinov

In Statistical and Nonlinear Physics

1. Ilya Kostrikov
2. Pratik Chaudhari
3. Jonas W. Mueller
4. Rasool Fakoor
5. Sergey Levine
6. Shixiang Shane Gu
7. Anssi Kanervisto
8. Alex Smola

In Reinforcement Learning

1. Sergey Levine
2. Shixiang Shane Gu
3. Ilya Kostrikov
4. Ashvin Nair
5. Chelsea Finn
6. Pieter Abbeel
7. Pratik Chaudhari
8. Alex Smola

Citations in the corpus

• Offline Reinforcement Learning with Implicit Q-Learning (2021, methodology): “In particular, quantile regression (Koenker & Hallock, 2001) has been previously used in reinforcement learning to estimate the quantile function of a state-action value function (Dabney et al., 2018b;a; Kuznetsov et al., 2020).”
• Continuous Doubly Constrained Batch Reinforcement Learning (2021, methodology): “Several strategies address overestimation [14, 16, 21, 22, 29, 32].”
• Randomized Ensembled Double Q-Learning: Learning Fast Without a Model (2021, methodology): “…deterministic policies at early stage of training (Fox et al., 2015), using multi-step methods (Meng et al., 2020), performing weighted Bellman updates to mitigate error propagation (Lee et al., 2020), and truncating sampled Q estimates with distributional networks (Kuznetsov et al., 2020).”
• Stable-Baselines3: Reliable Reinforcement Learning Implementations (2021, background): “This allows SB3 to maintain a stable and compact core, while still providing the latest features, like Truncated Quantile Critics (Kuznetsov et al., 2020).”
• Co-Adaptation of Algorithmic and Implementational Innovations in Inference-based Deep Reinforcement Learning (2021, methodology): “We center our analyses around MPO, AWR, and SAC, because they are representative algorithms that span both EM-based [49, 50, 43, 44, 1, 56, 42] and KL-control-based [59, 51, 12, 29, 21, 33, 32] RL and achieve some of the most competitive performances on popular benchmarks [6, 58].”
• IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies (2023, unknown): “Quantile statistics have also been used in RL prior for estimating distributions [9, 29] over Q -functions.”