Munchausen Reinforcement Learning

By Nino Vieillard, Olivier Pietquin, Matthieu Geist. 2020

116 citations recorded by the source corpus.

Bootstrapping is a core mechanism in Reinforcement Learning (RL). Most algorithms, based on temporal differences, replace the true value of a transiting state by their current estimate of this value. Yet, another estimate could be leveraged to bootstrap RL: the current policy. Our core contribution stands in a very simple idea: adding the scaled log-policy to the immediate reward. We show that slightly modifying Deep Q-Network (DQN) in that way provides an agent that is competitive with distributional methods on Atari games, without making use of distributional RL, n-step returns or prioritized replay. To demonstrate the versatility of this idea, we also use it together with an Implicit Quantile Network (IQN). The resulting agent outperforms Rainbow on Atari, installing a new State of the Art with very little modifications to the original algorithm. To add to this empirical study, we provide strong theoretical insights on what happens under the hood -- implicit Kullback-Leibler regularization and increase of the action-gap.

Subfields: Economics and Econometrics, Safety Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Economics and Econometrics

1. Sergey Levine
2. Pablo Samuel Castro
3. Rishabh Agarwal
4. Honglak Lee
5. Olivier Pietquin
6. Nando de Freitas
7. Aaron C. Courville
8. Marc G. Bellemare

In Safety Research

1. Honglak Lee
2. Sergey Levine
3. Quoc V. Le

Citations in the corpus

• Acme: A Research Framework for Distributed Reinforcement Learning (2020, background): “Munchausen DQN Discrete Actions Off-Policy Q-Network Bootstrapping Munchausen DQN (MDQN) (Vieillard et al., 2020) is an explicitly entropy-regularized and implicitly KLregularized version of DQN.”
• Evolving Reinforcement Learning Algorithms (2021, methodology): “M-DQN modifies DQN by adding the scaled log-policy (using the softmax Q-values) to the immediate reward.”
• Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “[109] Nino Vieillard, Olivier Pietquin, and Matthieu Geist.”
• Continuous Control with Action Quantization from Demonstrations (2021, methodology): “The RL algorithm considered is Munchausen DQN (Vieillard et al., 2020) as it is the state of the art on the Atari benchmark (Bellemare et al.”