Temporal Difference Variational Auto-Encoder

By Karol Gregor, Frederic Besse. 2018

132 citations recorded by the source corpus.

To act and plan in complex environments, we posit that agents should have a mental simulator of the world with three characteristics: (a) it should build an abstract state representing the condition of the world; (b) it should form a belief which represents uncertainty on the world; (c) it should go beyond simple step-by-step simulation, and exhibit temporal abstraction. Motivated by the absence of a model satisfying all these requirements, we propose TD-VAE, a generative sequence model that learns representations containing explicit beliefs about states several steps into the future, and that can be rolled out directly without single-step transitions. TD-VAE is trained on pairs of temporally separated time points, using an analogue of temporal difference learning used in reinforcement learning.

Subfields: Reinforcement Learning, Generative Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Reinforcement Learning

In Generative Models

Citations in the corpus

Learning Latent Dynamics for Planning from Pixels (2018, background): “…al., 2018; Srinivas et al., 2018), and pure video prediction without planning (Oh et al., 2015; Krishnan et al., 2015; Karl et al., 2016; Chiappa et al., 2017; Babaeizadeh et al., 2017; Gemici et al., 2017; Denton and Fergus, 2018; Buesing et al., 2018; Doerr et al., 2018; Gregor and Besse, 2018).”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “If the dynamics of the world are also learned, then it can be applied to planning [65, 39] or model-based RL [33, 50].”
Shaping Belief States with Generative Environment Models for RL (2019, background): “In this paper we refer to belief-states as any vector representation that is sufficient to predict future observations as in [9, 10].”
Mastering Atari with Discrete World Models (2020, methodology): “…al., 2016; Chiappa et al., 2017; Babaeizadeh et al., 2017; Gemici et al., 2017; Denton and Fergus, 2018; Buesing et al., 2018; Doerr et al., 2018; Gregor and Besse, 2018), or combine the benefits of model-based and model-free approaches (Kalweit and Boedecker, 2017; Nagabandi et al., 2017; Weber…”
Interpretable Visual Reasoning via Probabilistic Formulation Under Natural Supervision (2020, background): “Recent works TD-VAE [17] and PlaNet [19] further extend to time-sequential planning to solve more complex problems with sequential state observations.”
Keyframing the Future: Keyframe Discovery for Visual Prediction and Planning (2019, background): “Other recent work has considered predictions that “jump” more than one step into the future, using either ﬁxed-offset jumps (Buesing et al., 2018) or heuristics such as predictability of the frame (Neitz et al., 2018; Jayaraman et al., 2019; Gregor et al., 2019) to choose which frames to predict.”
Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Model (2019, methodology): “Prior work [8, 30, 21, 26, 20, 36, 12, 55] has explored modeling such non-Markovian observation sequences, using methods such as recurrent neural networks with deterministic hidden state, as well as probabilistic state-space models.”
Local Information Agent Modelling in Partially-Observable Environments (2021, methodology): “…improving the sample sample efﬁciency of RL algorithms using reconstruction-based [Corneil et al., 2018, Igl et al., 2018, Kurutach et al., 2018, Gregor et al., 2019, Gelada et al., 2019] or non-reconstruction-based methods [Laskin et al., 2020, Zhang et al., 2020], using contrastive learning…”
Simple and Effective VAE Training with Calibrated Decoders (2020, unknown): “These VAE techniques are used for tasks such as future frame prediction [7], image segmentation [28], generating sentences [6], speech [10], sketches [18], and music [12], as well as model-based reinforcement learning [19], and representation learning [31, 16, 46].”
Learning Robot Skills with Temporal Variational Inference (2020, background): “Kim et al. (2019) seeks to learn a generic partitioning of the input sequence into temporal abstractions, while Gregor et al. (2019) learns temporal abstractions with beliefs over state to capture uncertainity about the world.”