The Value Function Polytope in Reinforcement Learning
By Robert Dadashi, Adrien Ali Taiga, Nicolas Le Roux, Dale Schuurmans, Marc G. Bellemare. 2019
57 citations recorded by the source corpus.
We establish geometric and topological properties of the space of value functions in finite state-action Markov decision processes. Our main contribution is the characterization of the nature of its shape: a general polytope (Aigner et al., 2010). To demonstrate this result, we exhibit several properties of the structural relationship between policies and value functions including the line theorem, which shows that the value functions of policies constrained on all but one state describe a line segment. Finally, we use this novel perspective to introduce visualizations to enhance the understanding of the dynamics of reinforcement learning algorithms.
Subfields: Artificial Intelligence, Reinforcement Learning, Statistical Learning Theory
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Pieter Abbeel
- Yoshua Bengio
- Chelsea Finn
- Marc G. Bellemare
- Ruslan Salakhutdinov
- Mohammad Norouzi
- Ofir Nachum
In Reinforcement Learning
- Sergey Levine
- Pieter Abbeel
- Marc G. Bellemare
- Chelsea Finn
- Ofir Nachum
- Dale Schuurmans
- Mohammad Norouzi
- Timothy P. Lillicrap
In Statistical Learning Theory
- Pieter Abbeel
- Dale Schuurmans
- Sergey Levine
- Marc G. Bellemare
- Razvan Pascanu
- Will Dabney
- Tor Lattimore
- Anusha Nagabandi
Citations in the corpus
- Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, background): “It is plausible that following the natural gradient [1, 28] helps as well, see for example [4] (their figures 9b and 11b) for prelimi-nary evidence.”
- A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, background): “…most policies are uninteresting and hence may distract the representation learning process – we show, us-ing a recently developed geometric results (Dadashi et al., 2019), that the optimization process can in fact be restricted to the value functions of a special subset of deterministic policies…”
- Policy Evaluation Networks (2020, unknown): “Moreover, because the MDP is so small, we can visualize the value polytope (Dadashi et al., 2019), as seen in Figure 2a.”
- Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model (2019, background): “[10] R.”