CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation

By S. Gadre, Mitchell Wortsman, Gabriel Ilharco, Ludwig Schmidt, Shuran Song. Computer Vision and Pattern Recognition, 2022

324 citations recorded by the source corpus.

For robots to be generally useful, they must be able to find arbitrary objects described by people (i.e., be language-driven) even without expensive navigation training on in-domain data (i.e., perform zero-shot inference). We explore these capabilities in a unified setting: language- driven zero-shot object navigation (L-ZSON). Inspired by the recent success of open-vocabulary models for image classification, we investigate a straightforward framework, CLIP on Wheels (CoW), to adapt open-vocabulary models to this task without fine-tuning. To better evaluate L-ZSON, we introduce the Pasturebenchmark, which considers finding uncommon objects, objects described by spatial and appearance attributes, and hidden objects described relative to visible objects. We conduct an in-depth empirical study by directly deploying 22 CoW baselines across Habitat, Robothor,and Pasture. In total, we evaluate over 90k navigation episodes and find that (1) CoW baselines often struggle to leverage language descriptions but are proficient at finding uncommon objects. (2) A simple Co W, with CLIP-based object localization and classical exploration-and no additional training-matches the navigation efficiency of a state-of-the-art ZSON method trained for 500M steps on HabitatMp3d data. This same CoW provides a 15.6 percentage point improvement in success over a state-of-the-art ROBOTHOR ZSON model.11For code, data, and videos, see cow.cs.columbia.edu/

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/742b195fb4c2868a4e60012c8e0bf7db43bb5650

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “In [143], the authors introduce a zero-shot object navigation setting that uses an open-vocabulary classiﬁer such as CLIP [4] to compute the cosine similarity between an image and a user-speciﬁed description.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “Consequently, numerous methods are devoted to improving ( x i ,y i ) : image-text pair (Used by CLIPort [43], EmbCLIP [44], and CoW [45]) R3M [39] ResNet-50[16] Time-CL x i : i -th video frame, i j k Sim-Mani : Meta-World, Franka Kitchen, Adroit MVP [41] ViT-BViT-L MAE [46] Ni =1 − log P x i |…”
ChatGPT for Robotics: Design Principles and Model Abilities (2023, background): “These systems are often coupled with additional features from pretrained large-scale vision and language models models [11], [14], [15], [16], [37], [38], [39].”
VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation (2023, methodology): “CLIP on Wheels (CoW) [1] adopts a straightforward approach in which the robot explores the closest frontier until the target object is detected using either CLIP [24] features or an open-vocabulary object detector.”
CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models (2025, unknown): “…works utilize VLMs as intermediate components for perception and control, leveraging their strong semantic understanding and reasoning capabilities to decompose complex tasks [18, 23, 31, 50], detect objects [16, 22], or generate dense rewards [11, 37, 68] or goals [2, 9, 13, 41, 42, 49, 65, 74].”
OpenVLA: An Open-Source Vision-Language-Action Model (2024, background): “Vision-Language-Action Models A number of works have explored the use of VLMs for robotics, e.g., for visual state representations [12, 13], object detection [67], high-level planning [16], and for providing a feedback signal [68–71].”
Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success (2025, unknown): “…and vision foundation models to enhance robotic capabilities, utilizing them as pretrained visual representations that accelerate robotic policy learning [29, 32, 30, 19, 31], for object localization in robotics tasks [9, 45], and for high-level planning and reasoning [1, 17, 42, 16, 43, 18, 6].”