A Geometric Perspective on Optimal Representations for Reinforcement Learning

By Marc G. Bellemare, Will Dabney, Robert Dadashi, Adrien Ali Taiga, Pablo Samuel Castro, Nicolas Le Roux, Dale Schuurmans, Tor Lattimore, Clare Lyle. 2019

99 citations recorded by the source corpus.

We propose a new perspective on representation learning in reinforcement learning based on geometric properties of the space of value functions. We leverage this perspective to provide formal evidence regarding the usefulness of value functions as auxiliary tasks. Our formulation considers adapting the representation to minimize the (linear) approximation of the value function of all stationary policies for a given environment. We show that this optimization reduces to making accurate predictions regarding a special class of value functions which we call adversarial value functions (AVFs). We demonstrate that using value functions as auxiliary tasks corresponds to an expected-error relaxation of our formulation, with AVFs a natural candidate, and identify a close relationship with proto-value functions (Mahadevan, 2005). We highlight characteristics of AVFs and their usefulness as auxiliary tasks in a series of experiments on the four-room domain.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

Citations in the corpus

DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Recently, Bellemare et al. (2019); Dadashi et al. (2019) approached the representation learning problem from the perspective that a good representation is one that allows the prediction via a linear map of any value function in the value function space.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Moreover, there might be implications of misalignment between auxiliary tasks and the main task at hand (Bellemare et al., 2019).”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, background): “Also, one can view L ( θ ) as an inﬁnite set of Bellman optimality constraints, and each constraint provides an auxiliary objective for training [26, 5].”
A survey and critique of multiagent deep reinforcement learning (2018, background): “However, the idea of multiple predictions about arbitrary signals was originally suggested for RL, in the context of general value functions [108, 20] and there still open problems, for example, better theoretical understanding [107, 276].”
The Value Equivalence Principle for Model-Based Reinforcement Learning (2020, methodology): “As discussed, one possible strategy to define V is to use actual value functions in an attempt to span as much as possible of the value polytope V̈ [5].”
The Value Function Polytope in Reinforcement Learning (2019, background): “There is a connection between representation learning and the polytopal structure of value functions that we have started exploring (Bellemare et al., 2019).”
Applications of reinforcement learning in energy systems (2021, methodology): “Most current deep RL methods augment their main objective with additional losses, typically facilitating and regularizing the representation learning process [177 – 180].”
Discovery of Useful Questions as Auxiliary Tasks (2019, background): “A geometrical perspective on auxiliary tasks was introduced by Bellemare et al. (2019).”