Learning Awareness Models

By Brandon Amos, Laurent Dinh, Serkan Cabi, Thomas Rothörl, Sergio Gomez Colmenarejo, Alistair Muldal, Tom Erez, Yuval Tassa, Nando de Freitas, Misha Denil. 2018

49 citations recorded by the source corpus.

We consider the setting of an agent with a fixed body interacting with an unknown and uncertain external world. We show that models trained to predict proprioceptive information about the agent's body come to represent objects in the external world. In spite of being trained with only internally available signals, these dynamic body models come to represent external objects through the necessity of predicting their effects on the agent's own body. That is, the model learns holistic persistent representations of objects in the world, even though the only training signals are body signals. Our dynamics model is able to successfully predict distributions over 132 sensor readings over 100 steps into the future and we demonstrate that even when the body is no longer in contact with an object, the latent variables of the dynamics model continue to represent its shape. We show that active data collection by maximizing the entropy of predictions about the body---touch sensors, proprioception and vestibular information---leads to learning of dynamic models that show superior performance when used for control. We also collect data from a real robotic hand and show that the same models can be used to answer questions about properties of objects in the real world. Videos with qualitative results of our models are available at this https URL.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, methodology): “An even more cost efﬁcient approach would be to not refresh LSTM states at all.”
Learning Latent Dynamics for Planning from Pixels (2018, methodology): “Amos et al. (2018) train a deterministic dynamics model using observation overshooting for active exploration with a robotics hand.”
Causally Correct Partial Models for Reinforcement Learning (2020, background): “These challenges have motivated researchers to consider simpler models, henceforth referred to as partial models , i.e. models which are neither conditioned on, nor generate the full set of observed data (Oh et al., 2017; Amos et al., 2018; Guo et al., 2018; Gregor et al., 2019).”
Shaping Belief States with Generative Environment Models for RL (2019, background): “Generative environment models trained with overshooting have been explored in the context of model-based planning [31, 32, 33, 34].”
Graph networks as learnable physics engines for inference and control (2018, neutral): “One of our key contributions is an approach for learning physical dynamics models (Grzeszczuk et al., 1998; Fragkiadaki et al., 2015; Battaglia et al., 2016; Chang et al., 2016; Wat-3 ters et al., 2017; Ehrhardt et al., 2017; Amos et al., 2018).”
Neural Predictive Belief Representations (2018, methodology): “another direction, the three methods we considered can go beyond predicting only visual observations to other modalities of sensory inputs, such as proprioception and touch sensors (Amos et al., 2018).”
Planning to Explore via Self-Supervised World Models (2020, methodology): “Certain prior work has considered model-based exploration (Amos et al., 2018; Sharma et al., 2019), but was not shown to scale to complex visual observations, only using proprioceptive information.”
Mind's Eye: Grounded Language Model Reasoning through Simulation (2022, background): “World models should be capable of planning (Li et al., 2022a), predicting (Amos et al., 2018 2022)), while our focus is grounded reasoning in the form of natural language.”
World Discovery Models (2019, unknown): “We also can go beyond predicting only visual observations to other modalities of sensory inputs, such as proprioception and touch sensors (Amos et al., 2018).”
Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning (2020, methodology): “, 2020) and as the basis for search models in planning agents (Oh et al., 2017; Amos et al., 2018; Hafner et al., 2019; Schrittwieser et al., 2019).”
What can I do here? A Theory of Affordances in Reinforcement Learning (2020, background): “Much of the work on partial models (Oh et al., 2017; Amos et al., 2018; Guo et al., 2018; Gregor et al., 2019) focuses on models that predict only some of the state variables, or that make latent-space predictions (Schrittwieser et al., 2019), rather than restricting the space of actions considered…”
Temporal Difference Variational Auto-Encoder (2018, methodology): “Various methods have been used to alleviate this issue (Bengio et al., 2015; Lamb et al., 2016; Goyal et al., 2017; Amos et al., 2018).”