Learning Actionable Representations from Visual Observations

By Debidatta Dwibedi, Jonathan Tompson, Corey Lynch, Pierre Sermanet. 2018

84 citations recorded by the source corpus.

In this work we explore a new approach for robots to teach themselves about the world simply by observing it. In particular we investigate the effectiveness of learning task-agnostic representations for continuous control tasks. We extend Time-Contrastive Networks (TCN) that learn from visual observations by embedding multiple frames jointly in the embedding space as opposed to a single frame. We show that by doing so, we are now able to encode both position and velocity attributes significantly more accurately. We test the usefulness of this self-supervised approach in a reinforcement learning setting. We show that the representations learned by agents observing themselves take random actions, or other agents perform tasks successfully, can enable the learning of continuous control policies using algorithms like Proximal Policy Optimization (PPO) using only the learned embeddings as input. We also demonstrate significant improvements on the real-world Pouring dataset with a relative error reduction of 39.4% for motion attributes and 11.1% for static attributes compared to the single-frame baseline. Video results are available at https://sites.google.com/view/actionablerepresentations

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Representations by Maximizing Mutual Information Across Views (2019, background): “Self-supervised learning is gaining popularity across the NLP, vision, and robotics communities – e.g., [Devlin et al., 2019, Logeswaran and Lee, 2018, Sermanet et al., 2017, Dwibedi et al., 2018].”
Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixels (2020, background): “, 2018) or by employing auxiliary losses that provide additional supervision (Yarats et al., 2019; Srinivas et al., 2020; Sermanet et al., 2018; Dwibedi et al., 2018).”
CURL: Contrastive Unsupervised Representations for Reinforcement Learning (2020, methodology): “Contrastive learning has been used to extract reward signals in the latent space (Sermanet et al., 2018; Dwibedi et al., 2018; WardeFarley et al., 2018); and study representation learning on Atari games by Anand et al. (2019).”
Towards Effective Context for Meta-Reinforcement Learning: an Approach based on Contrastive Learning (2020, methodology): “In RL, it has been used to extract reward signals in latent space (Sermanet et al. 2018; Dwibedi et al. 2018), or used as an auxiliary task to study representations for high-dimensional data (Srinivas, Laskin, and Abbeel 2020; Anand et al.”
Graph-Structured Visual Imitation (2019, background): “Numerous objectives have been proposed to learn full image or image sequence convolutional image embeddings, such as multiview invariant and time-contrastive objectives in [2, 20], forward and inverse dynamics model learning in [4, 21], or reconstruction and temporal prediction objectives in [22, 23”
Value Function Spaces: Skill-Centric State Abstractions for Long-Horizon Reasoning (2021, background): “Contrastive learning has also been used in recent works to discover a meaningful latent space and extract reward signals for RL (Sermanet et al., 2017; Warde-Farley et al., 2018; Dwibedi et al., 2018; Laskin et al., 2020).”
Wasserstein Dependency Measure for Representation Learning (2019, background): “Self-supervised learning techniques have demonstrated state-of-the-art performance in speech and image understanding [47, 24], reinforcement learning [25, 18, 26], imitation learning [45, 5], and natural language processing [12, 43].”
Counting Out Time: Class Agnostic Video Repetition Counting in the Wild (2020, background): “The temporal context helps modeling short-term motion [13, 48] and enables the model to distinguish between similar looking frames but with different motion (e.”
Improving Sample Efficiency in Model-Free Reinforcement Learning from Images (2019, methodology): “Prior work has attempted to learn state representations from pixels with autoencoders, utilizing a two-step training procedure, where the representation is first trained via the autoencoder, and then either with a policy learned on top of the fixed representation (Lange and Riedmiller 2010; Munk, Ko”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “Prior work learns dynamics models for behavior cloning [56, 66, 43, 53, 52] or crafts reward functions that encourage visiting expert states [1, 16, 54, 35].”
Recent Advances in Imitation Learning from Observation (2019, background): “Dwibedi et al. [2018] claims that, since TCN uses single frames to learn the embedding function, it is difﬁcult for TCN to encode motion cues or the velocities of objects.”