CLIPort: What and Where Pathways for Robotic Manipulation

By Mohit Shridhar, Lucas Manuelli, Dieter Fox. 2021

963 citations recorded by the source corpus.

How can we imbue robots with the ability to manipulate objects precisely but also to reason about them in terms of abstract concepts? Recent works in manipulation have shown that end-to-end networks can learn dexterous skills that require precise spatial reasoning, but these methods often fail to generalize to new goals or quickly learn transferable concepts across tasks. In parallel, there has been great progress in learning generalizable semantic representations for vision and language by training on large-scale internet data, however these representations lack the spatial understanding necessary for fine-grained manipulation. To this end, we propose a framework that combines the best of both worlds: a two-stream architecture with semantic and spatial pathways for vision-based manipulation. Specifically, we present CLIPort, a language-conditioned imitation-learning agent that combines the broad semantic understanding (what) of CLIP [1] with the spatial precision (where) of Transporter [2]. Our end-to-end framework is capable of solving a variety of language-specified tabletop tasks from packing unseen objects to folding cloths, all without any explicit representations of object poses, instance segmentations, memory, symbolic states, or syntactic structures. Experiments in simulated and real-world settings show that our approach is data efficient in few-shot settings and generalizes effectively to seen and unseen semantic concepts. We even learn one multi-task policy for 10 simulated and 9 real-world tasks that is better or comparable to single-task policies.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, methodology): “Most of these prior works focus on following low-level instructions, such as for pick-and-place tasks and other robotic manipulation primitives [20, 22, 56, 21, 13, 26], though some methods address long-horizon, compound tasks in simulated domains [57, 58, 54].”
Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “CLIP has been employed in several robotics and embodied settings in zero-shot manner [74], or combined with Transporter networks [75] as in CLIPort [76].”
Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, methodology): “, models similar to CLIPort [83, 84] for open vocabulary pick-and-place).”
PaLM-E: An Embodied Multimodal Language Model (2023, background): “While many works focus on understanding natural language goals (Lynch & Sermanet, 2020; Shridhar et al., 2022a; Nair et al., 2022; Lynch et al., 2022), fewer consider natural language as a representation for planning – the focus of this work.”
Can Wikipedia Help Offline Reinforcement Learning? (2022, background): “, 2020), language for universal reward specification (Jiang et al., 2019; Lynch & Sermanet, 2021; Shridhar et al., 2022), and object-centric action spaces (Zeng et al.”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “grasping, pushing or placing) to each location [105, 124, 125].”
An Embodied Generalist Agent in 3D World (2023, positive): “To probe LEO’s capacity of bridging vision-language-acting in the 3D world, we select two canonical embodied AI tasks: embodied navigation ( ObjNav ) on AI Habitat (Ramrakhya et al., 2022) and robotic manipulation on CLIPort (Shridhar et al., 2021).”
Code as Policies: Language Model Programs for Embodied Control (2022, methodology): “To improve CLIPort performance, we use a variant that uses oracle information from the simulation to stop the policy when success is detected (oracle termination).”
Text2Motion: from natural language instructions to feasible plans (2023, background): “However, LCPs require expensive data collection and training procedures if they are to generalize to a wide distribution of long-horizon tasks with diverse instructions.”
Learning Universal Policies via Text-Guided Video Generation (2023, methodology): “While our environment setting is similar to that of (Shridhar et al., 2022), this method is not directly comparable to our approach, as CLIPort abstracts actions to the existing primitives of pick and place as opposed to using joint space of a robot.”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “To systematically evaluate agents with multimodal prompts, we develop a new benchmark, named VIMA-B ENCH , built on the Ravens simulator (Zeng et al., 2020; Shridhar et al., 2021).”
Correcting Robot Plans with Natural Language Feedback (2022, methodology): “Recently, instruction following has been studied in robotics by mapping instructions and raw visual observations to actions using end-to-end representation learning and sim-to-real transfer [8, 2, 35].”
SQA3D: Situated Question Answering in 3D Scenes (2022, background): “Robots now can navigate (Anderson et al., 2018) and manipulate objects (Liang et al., 2019; Savva et al., 2019; Shridhar et al., 2022; Ahn et al., 2022) following natural language commands Figure 2: Examples from SQA3D.”
Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation (2024, methodology): “…works also focus on multi-task or few-shot imitation learning, [25, 27, 30, 34, 46, 50, 88, 102], language-conditioned imitation learning [12, 47, 82, 83], imitation from play data [21, 57, 74, 89], using human videos [16, 24, 29, 60, 69, 80, 84, 96], and using task-specific structures [49, 83,…”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, background): “Although recent years have seen several large multi-task robot policies proposed in the literature [51, 23], such models often have limited breadth of real-world tasks, as with Gato [51], or focus on training tasks rather than generalization to new tasks, as with recent instruction following methods”
Instruction-driven history-aware policies for robotic manipulations (2022, methodology): “Instruction-driven vision-based robotic manipulation has received growing attention for manipulations in 2D planar [57, 58] or recent 3D environments [8, 59, 60], and has been transferred to the real world [9, 10].”
Open-Vocabulary Image Segmentation (2021, background): “Such interface may facilitate downstream applications such as interactive intelligent home assistants, interactive content creation, and instructing robot actions with language [41].”