Learning Latent Plans from Play

By Corey Lynch, Mohi Khansari, Ted Xiao, Vikash Kumar, Jonathan Tompson, Sergey Levine, Pierre Sermanet. 2019

512 citations recorded by the source corpus.

We propose learning from teleoperated play data (LfP) as a way to scale up multi-task robotic skill learning. Learning from play (LfP) offers three main advantages: 1) It is cheap. Large amounts of play data can be collected quickly as it does not require scene staging, task segmenting, or resetting to an initial state. 2) It is general. It contains both functional and non-functional behavior, relaxing the need for a predefined task distribution. 3) It is rich. Play involves repeated, varied behavior and naturally leads to high coverage of the possible interaction space. These properties distinguish play from expert demonstrations, which are rich, but expensive, and scripted unattended data collection, which is cheap, but insufficiently rich. Variety in play, however, presents a multimodality challenge to methods seeking to learn control on top. To this end, we introduce Play-LMP, a method designed to handle variability in the LfP setting by organizing it in an embedding space. Play-LMP jointly learns 1) reusable latent plan representations unsupervised from play data and 2) a single goal-conditioned policy capable of decoding inferred plans to achieve user-specified tasks. We show empirically that Play-LMP, despite not being trained on task-specific data, is capable of generalizing to 18 complex user-specified manipulation tasks with average success of 85.5%, outperforming individual models trained on expert demonstrations (success of 70.3%). Furthermore, we find that play-supervised models, unlike their expert-trained counterparts, 1) are more robust to perturbations and 2) exhibit retrying-till-success. Finally, despite never being trained with task labels, we find that our agent learns to organize its latent plan space around functional tasks. Videos of the performed experiments are available at learning-from-play.github.io

Subfields: Artificial Intelligence, Control and Systems Engineering, Social Psychology, Reinforcement Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Social Psychology

In Reinforcement Learning

In Robotics & Control

Citations in the corpus

Solving Rubik's Cube with a Robot Hand (2019, background): “Some researchers have successfully utilized expert human demonstrations in guiding the training process of the agents [17, 63, 4, 122].”
D4RL: Datasets for Deep Data-Driven Reinforcement Learning (2020, background): “…from a randomly sampled start location ( antmaze-umaze-diverse-v0 , antmaze-medium-diverse-v0 , and antmaze-large-diverse-v0 ), and a “play” dataset Lynch et al. [2019] where the ant is commanded speciﬁc hand-picked locations in the maze, starting from a diﬀerent set of hand-picked start locations…”
Learning to Generalize Across Long-Horizon Tasks from Human Demonstrations (2020, background): “due to the often complex correlation between behavior and task specification, these methods require large amounts of annotated demonstrations [25] and consequently, they do not scale well to physical robots in the real world.”
RoboNet: Large-Scale Multi-Robot Learning (2019, methodology): “Further, we also study how we can extend vision-based inverse models [10, 27, 6, 11] for generalizable robot-agnostic control.”
Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learning (2019, methodology): “RIL is a simple imitation learning procedure that builds on the goal relabeling scheme described in Lynch et al. [22] for the hierarchical setting, resulting in improved handling of multi-task generalization and compounding error.”
OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning (2020, methodology): “The above form of KL constraint is inspired from past works (Lynch et al., 2020; Kumar et al., 2020a).”
Accelerating Reinforcement Learning with Learned Skill Priors (2020, background): “via human teleoperation [28, 9, 20, 19] or any combination of these.”
IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Data (2019, methodology): “conditioned RNN πθ (s |g) (similar to [24]) trained on trajectory sequences of length T .”
Training Agents using Upside-Down Reinforcement Learning (2019, methodology): “Ghosh et al. [2019] studied an algorithm that is similar to the one by Lynch et al. [2020] in using pure SL and conditioning actions on fixed goal states, and similar to ours in that all data for learning is generated by the agent itself.”
Search on the Replay Buffer: Bridging Planning and Reinforcement Learning (2019, background): “Moreover, goal-conditioned RL often requires large amounts of reward shaping (Chiang et al., 2019) or human demonstrations (Lynch et al., 2019; Nair et al., 2018), both of which can limit the asymptotic performance of the policy by discouraging the policy from seeking novel solutions.”
Language Conditioned Imitation Learning Over Unstructured Data (2020, methodology): “Text conditioning is also a new and difﬁcult setting for goal-directed imitation learning [26, 8], which introduces important new research questions.”
Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skills (2020, methodology): “Note that unlike approaches in imitation learning (Ho & Ermon, 2016), learning from demonstrations (Hester et al., 2017; Veˇcer´ık et al., 2017), and learning from play (Lynch et al., 2019), EDL does not require access to trajectories or actions emitted by an expert policy.”
Hierarchical Planning for Long-Horizon Manipulation with Geometric and Symbolic Scene Graphs (2020, methodology): “To mitigate this problem, hierarchical structures have been introduced to neural network methods [8, 22, 24, 25, 27].”
DexPilot: Vision-Based Teleoperation of Dexterous Robotic Hand-Arm System (2019, background): “Additionally, the data can be regarded as play data which is useful to learn long-range planning [36].”
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies (2023, methodology): “We compare BESO against several state-of-the-art methods: • Goal-conditioned Behavior Cloning (GCBC) learns a unimodal policy encoded as a simple multi-layer perceptron (MLP) with an trained with an MSE loss [21].”
RvS: What is Essential for Offline RL via Supervised Learning? (2021, methodology): “While unimodal Gaussians are a common choice with continuous action spaces [14, 26, 27], a categorical distribution over a discretized action space allows the policy to represent more complex, multi-modal distributions [18, 29].”
Imitation Learning: A Survey of Learning Methods, Environments and Metrics (2024, unknown): “An example is the work from Lynch et al. [11], where the authors use behavioural cloning to condition a policy to a set of goals and, afterwards, use planning to create an optimal trajectory.”
Hierarchical Foresight: Self-Supervised Learning of Long-Horizon Tasks via Visual Subgoal Generation (2019, methodology): “Speciﬁcally, a simulated Franka Emika Panda robot arm is mounted in front of a desk (as used in (Lynch et al., 2019)).”
Interactive Language: Talking to Robots in Real Time (2022, background): “We sidestep both these scaling concerns by instead having operators continuously teleoperate long-horizon behaviors, with no requirements on low level task segmentation or resets [11], [25], [43] and”
V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning (2025, unknown): “Following many previous works in goal-conditioned robot manipulation (Finn and Levine, 2017; Lynch et al., 2020; Chebotar et al., 2021; Jang et al., 2022; Liu et al., 2022; Gupta et al., 2022), our current formulation of the optimization target assumes that we have access to visual goals.”