LanCon-Learn: Learning with Language to Enable Generalization in Multi-Task Manipulation
By Andrew Silva, Nina M. Moorman, William Silva, Zulfiqar Haider Zaidi, Nakul Gopalan, Matthew Craig Gombolay. 2021
40 citations recorded by the source corpus.
Robots must be capable of learning from previously solved tasks and generalizing that knowledge to quickly perform new tasks to realize the vision of ubiquitous and useful robot assistance in the real world. While multi-task learning research has produced agents capable of performing multiple tasks, these tasks are often encoded as one-hot goals. In contrast, natural language specifications offer an accessible means both for (1) users to describe a set of new tasks to the robot and (2) robots to reason about the similarities and differences among tasks through language-based task embeddings. Until now, multi-task learning with language has been limited to navigation based tasks and has not been applied to continuous manipulation tasks, requiring precision to grasp and move objects. We present LanCon-Learn, a novel attention-based approach to language-conditioned multi-task learning in manipulation domains to enable learning agents to reason about relationships between skills and task objectives through natural language and interaction. We evaluate LanCon-Learn for both reinforcement learning and imitation learning, across multiple virtual robot domains along with a demonstration on a physical robot. LanCon-Learn achieves up to a 200% improvement in zero-shot task success rate and transfers known skills to novel tasks faster than non-language-based baselines, demonstrating the utility of language for goal specification.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Li Fei-Fei
- Yoshua Bengio
- Quoc V. Le
- Trevor Darrell
- Ruslan Salakhutdinov
In Computer Vision and Pattern Recognition
- Sergey Levine
- Luc Van Gool
- Li Fei-Fei
- Trevor Darrell
- Dacheng Tao
- Jitendra Malik
- Andrew Zisserman
- Xiaogang Wang
In Reinforcement Learning
- Sergey Levine
- Chelsea Finn
- Pieter Abbeel
- Karol Hausman
- Pierre Sermanet
- Igor Mordatch
- Corey Lynch
- Brian Ichter
In Robotics & Control
- Sergey Levine
- Chelsea Finn
- Corey Lynch
- Pierre Sermanet
- Andy Zeng
- Tianhe Yu
- Julian Ibarz
- Brian Ichter
Citations in the corpus
- PaLM-E: An Embodied Multimodal Language Model (2023, background): “Prior works focus on combining vision and language inputs in an embodied setting with the goal of direct action prediction (Guhur et al., 2022; Shridhar et al., 2022b;a; Zhang & Chai, 2021; Silva et al., 2021; Jang et al., 2022; Nair et al., 2022; Lynch et al., 2022; Brohan et al., 2022).”
- Instruction-driven history-aware policies for robotic manipulations (2022, unknown): “LanCon-Learn [81] is a recent instruction-conditioned multi-task approach.”
- RT-1: Robotics Transformer for Real-World Control at Scale (2022, methodology): “As in RT-1, several works use language commands processed with Transformers as a robust framework for specifying and generalizing to new tasks (Zhang & Chai, 2021; Pashevich et al., 2021; Silva et al., 2021; Jang et al., 2021; Ahn et al., 2022; Nair et al., 2022).”
- 3D-VLA: A 3D Vision-Language-Action Generative World Model (2024, unknown): “For RLBench, we compare our model 3D-VLA with LanCon-Learn (Silva et al., 2021), which is a multi-task approach that can predict actions based on instruction-conditioned inputs.”
- Robotic Skill Acquisition via Instruction Augmentation with Vision-Language Models (2022, background): “Recent advances in deep learning with large amounts of data have led to advances in methods for learning instruction-conditioned policies [1, 28, 34, 43, 44].”