Fully Parameterized Quantile Function for Distributional Reinforcement Learning
By Derek Yang, Li Zhao, Zichuan Lin, Tao Qin, Jiang Bian, Tie-Yan Liu. 2019
183 citations recorded by the source corpus.
Distributional Reinforcement Learning (RL) differs from traditional RL in that, rather than the expectation of total returns, it estimates distributions and has achieved state-of-the-art performance on Atari Games. The key challenge in practical distributional RL algorithms lies in how to parameterize estimated distributions so as to better approximate the true continuous distribution. Existing distributional RL algorithms parameterize either the probability side or the return value side of the distribution function, leaving the other side uniformly fixed as in C51, QR-DQN or randomly sampled as in IQN. In this paper, we propose fully parameterized quantile function that parameterizes both the quantile fraction axis (i.e., the x-axis) and the value axis (i.e., y-axis) for distributional RL. Our algorithm contains a fraction proposal network that generates a discrete set of quantile fractions and a quantile value network that gives corresponding quantile values. The two networks are jointly trained to find the best approximation of the true distribution. Experiments on 55 Atari Games show that our algorithm significantly outperforms existing distributional RL algorithms and creates a new record for the Atari Learning Environment for non-distributed agents.
Subfields: Artificial Intelligence, Computational Theory and Mathematics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Olivier Pietquin
- Matthieu Geist
- Nino Vieillard
- Arsenii Kuznetsov
- Alexander Grishin
- Dmitry P. Vetrov
- Pavel Shvechikov
- Sergey Levine
In Computational Theory and Mathematics
- Nino Vieillard
- Arsenii Kuznetsov
- Olivier Pietquin
- Matthieu Geist
- Alexander Grishin
- Dmitry P. Vetrov
- Pavel Shvechikov
Citations in the corpus
- Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Critics (2020, methodology): “Yang et al. (2019) proposed a method to learn both quantile fractions and quantile values (i.e. locations and probabilities of elements in the mixture approximating the unknown distribution).”
- Munchausen Reinforcement Learning (2020, methodology): “To demonstrate the versatility of the M-RL principle, we also combine it with IQN [11], a recent and efficient distRL agent (note that IQN has had recent successors, such as Fully Parameterized Quantile Function (FQF) [35], to which in principle, we could also apply M-RL).”