Distributional reinforcement learning with linear function approximation
By Marc G. Bellemare, Nicolas Le Roux, Pablo Samuel Castro, Subhodeep Moitra. 2019
28 citations recorded by the source corpus.
Despite many algorithmic advances, our theoretical understanding of practical distributional reinforcement learning methods remains limited. One exception is Rowland et al. (2018)'s analysis of the C51 algorithm in terms of the Cramer distance, but their results only apply to the tabular setting and ignore C51's use of a softmax to produce normalized distributions. In this paper we adapt the Cramer distance to deal with arbitrary vectors. From it we derive a new distributional algorithm which is fully Cramer-based and can be combined to linear function approximation, with formal guarantees in the context of policy evaluation. In allowing the model's prediction to be any real vector, we lose the probabilistic interpretation behind the method, but otherwise maintain the appealing properties of distributional approaches. To the best of our knowledge, ours is the first proof of convergence of a distributional algorithm combined with function approximation. Perhaps surprisingly, our results provide evidence that Cramer-based distributional methods may perform worse than directly approximating the value function.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Mohammad Norouzi
- Pieter Abbeel
- Dale Schuurmans
- Ofir Nachum
- Rishabh Agarwal
- Yoshua Bengio
- Marc G. Bellemare
Citations in the corpus
- Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “Despite these algorithmic advances, our understanding of performance gains due to practical distributional RL methods remains limited [38, 6].”
- Learning to Score Behaviors for Guided Policy Optimization (2019, background): “These approaches have impressive experimental results [2, 8], with a growing body of theory [28, 25, 3, 29].”
- A Comparative Analysis of Expected and Distributional Reinforcement Learning (2019, methodology): “S51-lite, based on (Bellemare et al. 2019), outputs Z(x, a) as a categorical distribution whose probabilities are a linear function of φx,a and minimizes the squared Cramér distance.”