On Evaluation of Embodied Navigation Agents

By Peter Anderson, Angel X. Chang, Devendra Singh Chaplot, Alexey Dosovitskiy, Saurabh Gupta, Vladlen Koltun, Jana Kosecka, Jitendra Malik, Roozbeh Mottaghi, Manolis Savva and 1 more. 2018

1,027 citations recorded by the source corpus.

Skillful mobile operation in three-dimensional environments is a primary topic of study in Artificial Intelligence. The past two years have seen a surge of creative work on navigation. This creative output has produced a plethora of sometimes incompatible task definitions and evaluation protocols. To coordinate ongoing and future research in this area, we have convened a working group to study empirical methodology in navigation research. The present document summarizes the consensus recommendations of this working group. We discuss different problem statements and the role of generalization, present evaluation measures, and provide standard scenarios that can be used for benchmarking.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

On the Evaluation of Vision-and-Language Navigation Instructions (2021, background): “See Anderson et al. (2018a) and Ilharco et al. (2019) for details.”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “We include a Path Weighted version of both metrics that considers the length of the expert demonstration [2].”
Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, background): “The choice of GeometricGoal is deliberate – we aim to create the PointNav [11] equivalent for mobile manipulators.”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “SPL Success weighted by Path Length [1] trades-off SR against TL.”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, background): “Recent advancements at the intersection of Computer Vision, Natural Language Processing and Robotics have tried to bring together the understanding of the visual world and that of natural language from the perspective of robots and embodied agents [1], [2], [3], [4], [5].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “Examples of down-stream tasks that make use of visual exploration outputs (i.e. maps) include Image Navigation [26], [73], Point Navigation [11], [44] and Object Navigation [53], [54], [56].”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “This is same as the ObjectGoal task as described in Anderson et al. (2018a).”
Habitat: A Platform for Embodied AI Research (2019, methodology): “Specifically, in the context of point-goal navigation [2], we make two scientific contributions:”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, methodology): “2 Among those metrics, SPL is the recommended primary measure of navigation performance [1], as it considers both effectiveness and efﬁciency.”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, background): “– Success rate weighted by path length (SPL), as defined in [1], provides a measure of success normalized by the ratio between the length of the shortest path and the selected path.”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, methodology): “Navigation in the Room-to-Room task (Anderson et al., 2018b) requires an agent to find a route R (a sequence of viewpoints) from the start viewpoint S to the target viewpoint T according to the given instruction I.”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, background): “normalized and inversed. We measure the distance and length over the sequence of panoramas in the execution, and ignore changes of orientation. SED is related to success weighted by path length (SPL) [1], but is designed for instruction following in graphbased environments, where a speciﬁc correct p”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, methodology): “Compared to Anderson et al. (2018b), our annotation tool has additional capabilities including speech collection, virtual pose tracking, and timealignment between transcript and pose.”
The Replica Dataset: A Digital Replica of Indoor Spaces (2019, background): “…of replicas of real environments with powerful simulators such as AI Habitat [24] enables scalable machine learning that may yield models that can be directly deployed in the real world to perform tasks like embod-ied navigation [1], instruction following [2], and question answering [9].”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, methodology): “…task; navigation error , the average (shortest-path) distance be-tween the agent’s ﬁnal location and the nearest goal from that location; and SPL (Anderson et al., 2018a), which weights task success rate by travel distance as follows: where N is the number of tasks, S i indicates whether task i…”
ObjectNav Revisited: On Evaluation of Embodied Agents Navigating to Objects (2020, methodology): “[2] and a detailed description of the instantiation of these recommendations in challenges organized at the Embodied AI workshop at CVPR 2020 [1].”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, methodology): “Table 3: Performance comparison with the state of arts without beam search : Student-forcing (Anderson et al., 2018b), RPA (Wang et al., 2018b), and Speaker-Follower (Fried et al., 2018).”
VALAN: Vision and Language Agent Navigation (2019, methodology): “Metrics for assessing agents performance include: Navigation Error (NE) which is a measure of the distance between agent’s stopping location and the goal; Success Rate (SR) measures how often the agent stops within some acceptable distance of the goal; Success weighted by Path Length (SPL) [2] measu”
RVT: Robotic View Transformer for 3D Object Manipulation (2023, background): “Recently, vision-based policies [14, 15, 16, 17, 18, 19, 20, 21] have gained increasing attention since the high-dimensional visual sensory input provides more generalizable observation representation across tasks and is more accessible in real-world perception systems.”