γ-Models: Generative Temporal Difference Learning for Infinite-Horizon Prediction

By Michael Janner, Igor Mordatch, Sergey Levine. 2020

55 citations recorded by the source corpus.

We introduce the $\gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. Replacing standard single-step models with $\gamma$-models leads to generalizations of the procedures central to model-based control, including the model rollout and model-based value estimation. The $\gamma$-model, trained with a generative reinterpretation of temporal difference learning, is a natural continuous analogue of the successor representation and a hybrid between model-free and model-based mechanisms. Like a value function, it contains information about the long-term future; like a standard predictive model, it is independent of task reward. We instantiate the $\gamma$-model as both a generative adversarial network and normalizing flow, discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors, and empirically investigate its utility for prediction and control.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Statistical and Nonlinear Physics, Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sergey Levine
2. Benjamin Eysenbach
3. Michael Janner
4. Ruslan Salakhutdinov
5. Chelsea Finn
6. Qiyang Li
7. Pieter Abbeel
8. Yoshua Bengio

In Computer Vision and Pattern Recognition

1. Sergey Levine
2. Benjamin Eysenbach
3. Ruslan Salakhutdinov
4. Luc Van Gool
5. Chelsea Finn
6. Abhinav Kumar Gupta
7. Trevor Darrell
8. Jitendra Malik

In Statistical and Nonlinear Physics

1. Qiyang Li
2. Sergey Levine
3. Michael Janner
4. Ruslan Salakhutdinov
5. Tianjun Zhang
6. Benjamin Eysenbach
7. Vladlen Koltun

In Reinforcement Learning

1. Sergey Levine
2. Benjamin Eysenbach
3. Michael Janner
4. Qiyang Li
5. Ruslan Salakhutdinov
6. Chelsea Finn
7. Tianjun Zhang
8. Pieter Abbeel

In Generative Models

1. Sergey Levine
2. Qiyang Li
3. Michael Janner
4. Benjamin Eysenbach
5. Ruslan Salakhutdinov
6. Yoshua Bengio
7. Igor Mordatch
8. Aaron C. Courville

Citations in the corpus

• Offline Reinforcement Learning as One Big Sequence Modeling Problem (2021, methodology): “…most obvious in model-based RL, where it is used to train predictive models that can then be used for planning or policy learning (Sutton, 1990; Silver et al., 2008; Fairbank, 2008; Deisenroth & Rasmussen, 2011; Lampe & Riedmiller, 2014; Heess et al., 2015b; Janner et al., 2020; Amos et al., 2021).”
• Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “] is a prototypical actor-critic method that uses hindsight relabeling [6]; Goal-conditioned behavioral cloning (GCBC) [22, 41, 83, 117] performs behavior cloning on relabeled experience; model-based fits a density model to the discounted state occupancy measure, similar on [21, 23, 60].”