ZSON: Zero-Shot Object-Goal Navigation using Multimodal Goal Embeddings

By Arjun Majumdar, Gunjan Aggarwal, Bhavika Devnani, Judy Hoffman, Dhruv Batra. Neural Information Processing Systems, 2022

316 citations recorded by the source corpus.

We present a scalable approach for learning open-world object-goal navigation (ObjectNav) -- the task of asking a virtual robot (agent) to find any instance of an object in an unexplored environment (e.g.,"find a sink"). Our approach is entirely zero-shot -- i.e., it does not require ObjectNav rewards or demonstrations of any kind. Instead, we train on the image-goal navigation (ImageNav) task, in which agents find the location where a picture (i.e., goal image) was captured. Specifically, we encode goal images into a multimodal, semantic embedding space to enable training semantic-goal navigation (SemanticNav) agents at scale in unannotated 3D environments (e.g., HM3D). After training, SemanticNav agents can be instructed to find objects described in free-form natural language (e.g.,"sink","bathroom sink", etc.) by projecting language goals into the same multimodal, semantic embedding space. As a result, our approach enables open-world ObjectNav. We extensively evaluate our agents on three ObjectNav datasets (Gibson, HM3D, and MP3D) and observe absolute improvements in success of 4.2% - 20.0% over existing zero-shot methods. For reference, these gains are similar or better than the 5% improvement in success between the Habitat 2020 and 2021 ObjectNav challenge winners. In an open-world setting, we discover that our agents can generalize to compound instructions with a room explicitly mentioned (e.g.,"Find a kitchen sink") and when the target room can be inferred (e.g.,"Find a sink and a stove").

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4f0f2096e006725576f1ccf0edda7c234d38fa66

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

An Embodied Generalist Agent in 3D World (2023, unknown): “We consider both Habitat-web (Ramrakhya et al., 2022) (fully supervised) and ZSON (Majumdar et al., 2022) (zero-shot) as baselines.”
Open-vocabulary Queryable Scene Representations for Real World Planning (2022, methodology): “Recently, methods that leverage pre-trained image-text models can do zero-shot Object Goal Navigation [31], [32].”
CoWs on Pasture: Baselines and Benchmarks for Language-Driven Zero-Shot Object Navigation (2022, methodology): “Recent work studies object navigation in zero-shot settings, where agents are evaluated on object categories that they are not explicitly trained on [38,46].”
VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation (2023, methodology): “We evaluate VLFM by comparing it to several state-of-the-art (SOTA) techniques for zero-shot object navigation: CLIP on Wheels (CoW) [1], ESC [2], SemUtil [3], and ZSON [32].”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, methodology): “[68] use LLM to extract the commonsense knowledge of the relations between targets and objects in observations to perform zero-shot object navigation (ZSON) [16, 38].”
Multi-Scale Gaussian-Language Map for Zero-shot Embodied Navigation and Reasoning (2026, unknown): “Recently, large models have enabled zero-shot ObjectNav methods [9, 22, 57] that use them as exter-nal priors without task-specific training, which are categorized as LLM-, VLM-, or MLLM-based methods.”