ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects

By Dhruv Batra, Aaron Gokaslan, Aniruddha Kembhavi, Oleksandr Maksymets, Roozbeh Mottaghi, Manolis Savva, Alexander Toshev, Erik Wijmans. 2020

371 citations recorded by the source corpus.

We revisit the problem of Object-Goal Navigation (ObjectNav). In its simplest form, ObjectNav is defined as the task of navigating to an object, specified by its label, in an unexplored environment. In particular, the agent is initialized at a random location and pose in an environment and asked to find an instance of an object category, e.g., find a chair, by navigating to it. As the community begins to show increased interest in semantic goal specification for navigation tasks, a number of different often-inconsistent interpretations of this task are emerging. This document summarizes the consensus recommendations of this working group on ObjectNav. In particular, we make recommendations on subtle but important details of evaluation criteria (for measuring success when navigating towards a target object), the agent's embodiment parameters, and the characteristics of the environments within which the task is carried out. Finally, we provide a detailed description of the instantiation of these recommendations in challenges organized at the Embodied AI workshop at CVPR 2020 this http URL .

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Control and Systems Engineering, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

In Robotics & Control

Citations in the corpus

Continuous Scene Representations for Embodied AI (2022, background): “Recently, various embodied tasks such as navigation [5,8,37,54,56,63], instruction following [1,47], embodied question answering [15,23], object manipulation [18,21,22,46], and room rearrangement [4,52] have been of interest.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, background): “It is noteworthy that there are some known issues with success weighted by path length [38].”
MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge (2022, background): “Note that we categorize “Find Nether Portal” and “Find Ocean” as Creative tasks even though they seem similar to object navigation [12].”
Pushing it out of the Way: Interactive Visual Navigation (2021, background): “A popular task that has received a considerable amount of attention is visual navigation [3, 5, 8, 29, 39, 48], where the goal is to navigate towards a specific coordinate or object within an unseen environment.”
Habitat-Matterport 3D Dataset (HM3D): 1000 Large-scale 3D Environments for Embodied AI (2021, methodology): “We benchmark embodied agents on the PointGoal navigation (a.k.a. PointNav) task [7], which has served as a standard testbed for exploring ideas in navigation [29, 31–33] and a starting point for more semantic tasks [34–36].”
Where are we in the search for an Artificial Visual Cortex for Embodied Intelligence? (2023, methodology): “We consider two tasks in Habitat: image-goal navigation ( ImageNav ) [44] and object-goal navigation ( ObjectNav ) [45].”
VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation (2023, background): “We address the task of ObjectNav [10], where a robot is tasked with searching for an instance of a target object category ( e.g. , ‘bed’) in a previously unseen environment.”
Success Weighted by Completion Time: A Dynamics-Aware Evaluation Criteria for Embodied Navigation (2021, background): “This and other limitations of SPL are discussed in Batra et al [2].”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, background): “Consider that the inconsistent interpretations of the object navigation task are emerging among the increasing number of works, [Batra et al., 2020]”
Simple but Effective: CLIP Embeddings for Embodied AI (2021, background): “Such tasks, including Object Goal Navigation [3] and Room Rearrangement [2], are inherently navigation-heavy.”
Exploiting Scene-specific Features for Object Goal Navigation (2020, background): “The SPL is considered today as the main metric for the Object Navigation task, but as described in [3] there are numerous problems.”
Rearrangement: A Challenge for Embodied AI (2020, methodology): “…learning have proposed various standardized evaluation methods, including protocols for evaluating robotic grasping [52, 72, 49], navigation [1, 4], manipulation tasks focusing on standardized real-world object sets [13], and competition-style setups, such as RoboCup [43, 67] and the…”
Object Goal Navigation using Goal-Oriented Semantic Exploration (2020, methodology): “SemExp also won the CVPR 2020 Habitat ObjectNav Challenge.”
CLIP on Wheels: Zero-Shot Object Navigation as Object Localization and Exploration (2022, methodology): “• Success weighted by inverse path length ( SPL ): success is weighted by the oracle shortest path length and normalized by the actual path length [5].”
An Exploration of Embodied Visual Exploration (2020, positive): “2020) and object navigation (Batra et al. 2020) have been proposed with standardized benchmarks.”