Unsupervised Perceptual Rewards for Imitation Learning

By Pierre Sermanet, Kelvin Xu, Sergey Levine. 2016

167 citations recorded by the source corpus.

Reward function design and exploration time are arguably the biggest obstacles to the deployment of reinforcement learning (RL) agents in the real world. In many real-world tasks, designing a reward function takes considerable hand engineering and often requires additional sensors to be installed just to measure whether the task has been executed successfully. Furthermore, many interesting tasks consist of multiple implicit intermediate steps that must be executed in sequence. Even when the final outcome can be measured, it does not necessarily provide feedback on these intermediate steps. To address these issues, we propose leveraging the abstraction power of intermediate visual representations learned by deep models to quickly infer perceptual reward functions from small numbers of demonstrations. We present a method that is able to identify key intermediate steps of a task from only a handful of demonstration sequences, and automatically identify the most discriminative features for identifying these steps. This method makes use of the features in a pre-trained deep model, but does not require any explicit specification of sub-goals. The resulting reward functions can then be used by an RL agent to learn to perform the task in real-world settings. To evaluate the learned reward, we present qualitative results on two real-world tasks and a quantitative evaluation against a human-designed reward function. We also show that our method can be used to learn a real-world door opening skill using a real robot, even when the demonstration used for reward learning is provided by a human using their own hand. To our knowledge, these are the first results showing that complex robotic manipulation skills can be learned directly and without supervised labels from a video of a human performing the task. Supplementary material and data are available at this https URL

Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Control and Systems Engineering

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Visual Imitation Made Easy (2020, background): “There have also been recent efforts to imitate from visual demonstrations collected from a different space e.g. from a different viewpoint or an agent with a different embodiment [6, 11] from the robot.”
Self-Supervised Correspondence in Visuomotor Policy Learning (2019, background): “A more generalizable direction may be offered by unsupervised methods of obtaining reward signals [2], [17], [18].”
One-Shot Visual Imitation Learning via Meta-Learning (2017, background): “However, this requires additional robot experience to optimize the reward [19, 20, 21].”
An Algorithmic Perspective on Imitation Learning (2018, background): “Recent work by Sermanet et al. [2017] shows that the reward function can be inferred from few demonstrations by using visual representations learned by deep models.”
One-Shot Imitation from Observing Humans via Domain-Adaptive Meta-Learning (2018, background): “a reward function based on human demonstrations [47, 49, 26, 46, 50].”
Zero-Shot Visual Imitation (2018, background): “Sermanet et al. (2017; 2018) imitate humans with robots by self-supervised learning but require expert supervision at training time.”
Learning by Playing - Solving Sparse Reward Tasks from Scratch (2018, background): “…et al. (2017); Rusu et al. (2017) for recent examples), learning from demonstrations (Ross et al., 2011; Vecerik et al., 2017; Kober & Peters, 2011; Sermanet et al., 2017; Nair et al., 2017), learning with model guidance, see e.g. Montgomery & Levine (2016), or inverse RL (Ng & Russell, 2000;…”
Zero-Shot Reward Specification via Grounded Natural Language (2022, background): “There are works that use human videos to learn reward functions to train their agent with (Shao et al., 2020; Sermanet et al., 2018; 2016), but they require a curated dataset of humans performing the tasks.”
Real-World Robot Learning with Masked Visual Pre-training (2022, methodology): “…on self-collected trajectories [33, 34]; learning visual autoencoders on trajectories [35]; learning spatiotemporal representations through videos [36, 37]; learning visual correspondence [38]; utilizing non-parametric nearest-neighbor retrieval [39]; and conducting visual self-supervised…”
Human Hands as Probes for Interactive Object Understanding (2021, background): “For instance, distance in a state-sensitive feature space can be used as reward functions for manipulation tasks [52,54,64].”
Grounding Language in Play (2020, background): “Designing reward functions in open-world environments is non-trivial, requiring either task-speciﬁc instrumentation (Gu et al., 2017) or learned perceptual rewards (Sermanet et al., 2017; 2018; Singh et al., 2019).”
Imitating Latent Policies from Observation (2018, background): “, 2017) and unsupervised perceptual rewards (Sermanet et al., 2016) train robots to imitate from demonstrations of humans performing tasks, and recent work used audio to align different YouTube videos to train an agent to learn Montezuma’s revenge and Pitfall (Aytar et al.”
Time-Contrastive Networks: Self-Supervised Learning from Multi-view Observation (2017, background): “Imitation learning: In the context of robotics and reinforcement learning, prior works have also considered the problem of imitating a demonstration observed from a different viewpoint or from an agent with a different embodiment, such as a human [24, 6, 22].”
Open X-Embodiment: Robotic Learning Datasets and RT-X Models : Open X-Embodiment Collaboration0 (2023, unknown): “Al-ternatively, some works focus on sub-aspects of the problem such as learning transferable reward functions [17, 44–48], goals [49, 50], dynamics models [51], or visual representations [52–59] from human video data.”
One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL (2018, background): “First, it would be exciting to combine this work with existing methods for learning third-person imitation rewards (Sermanet et al., 2016; 2018; Aytar et al., 2018).”
AVID: Learning Multi-Stage Tasks via Pixel-Level Translation of Human Videos (2019, methodology): “To handle human videos, some prior meth-ods have used explicit pose and object detection [11]–[13], predictive modeling [14], [15], context translation [16], [17], and learning reward representations [18], [19].”
OptionGAN: Learning Joint Reward-Policy Options using Generative Adversarial Inverse Reinforcement Learning (2017, methodology): “Similarly, in (Sermanet, Xu, and Levine 2016), techniques in video editing based on information-similarity are adopted to divide a video demonstration into distinct sections which can then be recombined into a differentiable reward function.”
XIRL: Cross-embodiment Inverse Reinforcement Learning (2021, methodology): “, collected via teleoperation) [2, 3], much less work has been done in getting robots to learn policies from tasks defined only by third-person observations of demonstrations [4, 5, 6].”
Reward Estimation for Variance Reduction in Deep Reinforcement Learning (2018, background): “In some cases, such as in [6, 21], the reward function is modified to be compatible and beneficial to the learning process.”