Success Weighted by Completion Time: A Dynamics-Aware Evaluation Criteria for Embodied Navigation

By Naoki Yokoyama, Sehoon Ha, Dhruv Batra. 2021

41 citations recorded by the source corpus.

We present Success weighted by Completion Time (SCT), a new metric for evaluating navigation performance for mobile robots. Several related works on navigation have used Success weighted by Path Length (SPL) as the primary method of evaluating the path an agent makes to a goal location, but SPL is limited in its ability to properly evaluate agents with complex dynamics. In contrast, SCT explicitly takes the agent’s dynamics model into consideration, and aims to accurately capture how well the agent has approximated the fastest navigation behavior afforded by its dynamics. While several embodied navigation works use point-turn dynamics, we focus on unicycle-cart dynamics for our agent, which better exempli-fies the dynamics model of popular mobile robotics platforms (e.g., LoCoBot, TurtleBot, Fetch, etc.). We also present RRT*-Unicycle, an algorithm for unicycle dynamics that estimates the fastest collision-free path and completion time from a starting pose to a goal location in an environment containing obstacles. We experiment with deep reinforcement learning and reward shaping to train and compare the navigation performance of agents with different dynamics models. In evaluating these agents, we show that in contrast to SPL, SCT is able to capture the advantages in navigation speed a unicycle model has over a simpler point-turn model of dynamics. Lastly, we show that we can successfully deploy our trained models and algorithms outside of simulation in the real world. We embody our agents in a real robot to navigate an apartment, and show that they can generalize in a zero-shot manner. A video summary is available here: https://youtu.be/QOQ56XVIYVE

Subfields: Artificial Intelligence, Biomedical Engineering, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

In Robotics & Control

Citations in the corpus

Habitat 2.0: Training Home Assistants to Rearrange their Habitat (2021, methodology): “– Agent action space: Similarily to [100], the navigation is handeled by a discrete action space which is then translated into continuous actions.”
VLFM: Vision-Language Frontier Maps for Zero-Shot Semantic Navigation (2023, methodology): “Unlike ObjectNav, PointNav does not necessitate a semantic understanding of the environment for efficient and successful completion and can be accomplished using only geometric understandings.”