Goal-conditioned Imitation Learning

By Yiming Ding, Carlos Florensa, Mariano Phielipp, Pieter Abbeel. 2019

277 citations recorded by the source corpus.

Designing rewards for Reinforcement Learning (RL) is challenging because it needs to convey the desired task, be efficient to optimize, and be easy to compute. The latter is particularly problematic when applying RL to robotics, where detecting whether the desired configuration is reached might require considerable supervision and instrumentation. Furthermore, we are often interested in being able to reach a wide range of configurations, hence setting up a different reward every time might be unpractical. Methods like Hindsight Experience Replay (HER) have recently shown promise to learn policies able to reach many goals, without the need of a reward. Unfortunately, without tricks like resetting to points along the trajectory, HER might require many samples to discover how to reach certain areas of the state-space. In this work we investigate different approaches to incorporate demonstrations to drastically speed up the convergence to a policy able to reach any goal, also surpassing the performance of an agent trained with other Imitation Learning algorithms. Furthermore, we show our method can also be used when the available expert trajectories do not contain the actions, which can leverage kinesthetic or third person demonstration. The code is available at this https URL.

Subfields: Artificial Intelligence, Cognitive Neuroscience, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cognitive Neuroscience

In Control and Systems Engineering

Citations in the corpus

IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, background): “cent work has extended reinforcement learning [4, 27, 31] and imitation learning [7, 24] to condition on goal observations, enabling improved sample efficiency.”
Language-Conditioned Imitation Learning for Robot Manipulation Tasks (2020, methodology): “While we use the term goals , it may refer to end goals (e.g., target objects) or constraints on motion (e.g., minimizing end-point effector acceleration) [14].”
Language Conditioned Imitation Learning Over Unstructured Data (2020, background): “This technique trains goal conditioned policies to reach any previously visited state upon demand, with many recent examples in RL [29, 14, 28] and IL [26, 8].”
Grounding Language in Play (2020, methodology): “…technique trains goal conditioned policies to reach any previously visited state upon demand, with many recent examples in RL (Nair et al., 2018; Pong et al., 2018; Gupta et al., 2019; Pong et al., 2019; Ghosh et al., 2019; Eysenbach et al., 2020) and IL (Lynch et al., 2019; Ding et al., 2019).”
Learning to Reach Goals via Iterated Supervised Learning (2019, methodology): “By generating demonstrations using hindsight relabelling, we are able to apply goal-conditioned imitation learning primitives [Gupta et al., 2019, Ding et al., 2019] on data collected by sub-optimal agents, not just from an expert supervisor.”
Transformers for One-Shot Visual Imitation (2020, background): “There are myriad ways to introduce goal conditioning, including with the robot’s state [24], ”goal” images of the ﬁnal state [25, 26, 27], natural language [28], and video or images of humans [29, 30].”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, methodology): “By conditioning on some piece of information, such as a goal, reward function parameterization, or reward value, such experience can be used for simple behavior cloning [5, 6, 8, 10, 17, 23, 26, 29, 32, 34, 38].”
Learning To Reach Goals Without Reinforcement Learning (2019, methodology): “y optimizing a lower bound on the goal-reaching objective. While several prior works have proposed training goal-conditioned policies via imitation learning based on a superﬁcially similar algorithm (Ding et al., 2019; Lynch et al., 2019), to our knowledge no prior work proposes a complete policy le”
Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skills (2021, methodology): “Prior methods for goal-conditioned supervised learning (Oh et al., 2018; Ding et al., 2019; Ghosh et al., 2019; Sun et al., 2019; Lynch et al., 2019) have a similar motivation: learning “useful” information for control without access to rewards.”
Continuous Control with Action Quantization from Demonstrations (2021, background): “…together with a high-level controller that is either learned from demonstrations (Kroemer et al., 2015; Krishnan et al., 2017; Le et al., 2018; Ding et al., 2019; Lynch et al., 2020), or learned from interactions with the environment (Manschitz et al., 2015; Kipf et al., 2019; Shankar et al.,…”
Learning Deep Parameterized Skills from Demonstration for Re-targetable Visuomotor Control (2019, result): “Recent work [8] has successfully extended goal-parameterization to GAIL and shown encouraging results in simulation.”
3D Diffuser Actor: Policy Diffusion with 3D Scene Representations (2024, unknown): “Generative adversarial networks [27, 76, 16], variational autoencoders [47] and combined Categorical and Gaussian distributions [66, 23, 14] have been used to learn from multimodal demonstrations.”
Generalized Hindsight for Reinforcement Learning (2020, methodology): “Ding et al. (2019) uses hindsight relabeling as a form of data augmentation for a limited set of expert trajectories, which it then uses with adversarial imitation learning to learn goal-conditioned policies.”
Interactive Language: Talking to Robots in Real Time (2022, background): “To condition multiple learned behaviors, prior setups have relied on discrete one-hot task identifiers [23], which can be difficult to scale to many tasks, or goal images [24]–[26], which can be impractical”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, methodology): “Note that setting λ = 1 corresponds to GCBC [15, 21, 24, 40, 83, 96, 116, 119], which we will include as a baseline.”