Simple but Effective: CLIP Embeddings for Embodied AI

By Apoorv Khandelwal, Luca Weihs, Roozbeh Mottaghi, Aniruddha Kembhavi. 2021

294 citations recorded by the source corpus.

Contrastive language image pretraining (CLIP) encoders have been shown to be beneficial for a range of visual tasks from classification and detection to captioning and image manipulation. We investigate the effectiveness of CLIP visual backbones for Embodied AI tasks. We build incredibly simple baselines, named EmbCLIP, with no task specific architectures, inductive biases (such as the use of semantic maps), auxiliary tasks during training, or depth maps-yet we find that our improved baselines perform very well across a range of tasks and simulators. EmbCLIP tops the RoboTHOR ObjectNav leader-board by a huge margin of 20 pts (Success Rate). It tops the iTHOR 1-Phase Rearrangement leaderboard, beating the next best submission, which employs Active Neural Mapping, and more than doubling the % Fixed Strict metric (0.08 to 0.17). It also beats the winners of the 2021 Habitat ObjectNav Challenge, which employ auxiliary tasks, depth maps, and human demonstrations, and those of the 2019 Habitat PointNav Challenge. We evaluate the ability of CLIP's visual representations at capturing semantic information about input observations-primitives that are useful for navigation-heavy embodied tasks- and find that CLIP's representations encode these primitives more effectively than ImageNet-pretrained backbones. Finally, we extend one of our baselines, producing an agent capable of zero-shot object navigation that can navigate to objects that were not used as targets during training. Our code and models are available at https://github.com/allenai/embodied-clip.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Inner Monologue: Embodied Reasoning through Planning with Language Models (2022, methodology): “CLIP has been employed in several robotics and embodied settings in zero-shot manner [80], or combined with Transporter networks [81] as in CLIPort [82].”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, methodology): “Inspired by the advancements in self-supervised learning, recent work has incorporated visual representation learning into the training pipelines for EAI agents (Parisi et al., 2022; Nair et al., 2022; Radosavovic et al., 2022; Ma et al., 2022; Khandelwal et al., 2022; Yadav et al., 2022b; 2023).”
AnyMorph: Learning Transferable Polices By Inferring Agent Morphology (2022, background): “2019), GPT-3 (Brown et al., 2020), and CLIP (Radford et al., 2021) exemplify how large multi-task models zeroshot generalize in out-of-domain applications like CLIP in robotics (Khandelwal et al., 2021).”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Consequently, numerous methods are devoted to improving ( x i ,y i ) : image-text pair (Used by CLIPort [43], EmbCLIP [44], and CoW [45]) R3M [39] ResNet-50[16] Time-CL x i : i -th video frame, i j k Sim-Mani : Meta-World, Franka Kitchen, Adroit MVP [41] ViT-BViT-L MAE [46] Ni =1 − log P x i |…”
VIMA: General Robot Manipulation with Multimodal Prompts (2022, methodology): “, 2022) and Embodied-CLIP (Khandelwal et al., 2021) leverage the frozen visual and textual representations of CLIP (Radford et al.”
LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Action (2022, background): “…1M demonstrations — annotated with ﬁne-grained language labels in indoor [1, 3–5, 22 ] Recent progress in using large-scale models of natural language and images trained on diverse data has enabled applications in a wide variety of textual [31–33], visual [13, 34–38], and embodied domains [39–44].”
Semantic Exploration from Language Abstractions and Pretrained Representations (2022, background): “These features can condition the policy network [26], or can be fused throughout the visual encoder to integrate semantic information about the environment [37].”
R3M: A Universal Visual Representation for Robot Manipulation (2022, methodology): “Prior work has also found visual representations informed by language, like CLIP [12], to be effective for control [35, 36].”
ProcTHOR: Large-Scale Embodied AI Using Procedural Generation (2022, methodology): “For instance, in embodied AI, the houses may be used for navigation [58, 89, 118, 132, 117, 128, 74, 130], multi-agent interaction [51, 52, 2], rearrangement and interaction [115, 36, 39, 23, 106], manipulation [33, 86, 32, 122], Sim2Real transfer [27, 54, 66], embodied vision-andlanguage [105, 87, ”
Instruction-driven history-aware policies for robotic manipulations (2022, background): “Because natural language is rich and diverse, while training resources are limited, further works learn from collected offline data with instructions [10, 17] or leverage pre-trained vision-language models [4, 5] for action prediction [9, 68].”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “To further contextualize our results, we compare our CoW models to prior zero-shot object navigation work: the EmbCLIP zero-shot model [36].”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, methodology): “In this paper, we study Language-driven zero-shot object navigation (L-ZSON)—a more challenging but also more applicable version of object navigation [4,17,63,74,83] and ZSON [37, 44] tasks.”
Can Foundation Models Perform Zero-Shot Task Specification For Robot Manipulation? (2022, result): “The concurrent works of Khandelwal et al. (2021) and Parisi et al. (2022), study the use CLIP and other self-supervised representation networks as a perception module for control tasks and observe they outperform traditional ImageNet-pretrained backbones.”
ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings (2022, background): “Such a solution was explored in EmbCLIP [25] with promising results.”