Effective and General Evaluation for Instruction Conditioned Navigation using Dynamic Time Warping
By Gabriel Ilharco, Vihan Jain, Alexander Ku, Eugene Ie, Jason Baldridge. 2019
26 citations recorded by the source corpus.
Subfields: Evaluation & Benchmarks, Robotics & Control
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Evaluation & Benchmarks
- Mohit Bansal
- Dhruv Batra
- Stefan Lee
- Hao Hao Tan
- Jason Baldridge
- Erik Wijmans
- Eugene Ie
- Piotr Wojciech Mirowski
In Robotics & Control
- Dhruv Batra
- Peter Anderson
- Erik Wijmans
- Stefan Lee
- Mohit Bansal
- Arjun Majumdar
- Eugene Ie
- Jason Baldridge
Citations in the corpus
- On the Evaluation of Vision-and-Language Navigation Instructions (2021, neutral): “See Anderson et al. (2018a) and Ilharco et al. (2019) for details.”
- Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “As in Ilharco et al. (2019), the reward at each step is the incremental difference in NDTW, plus a linear function of navigation error after stopping.”
- VALAN: Vision and Language Agent Navigation (2019, methodology): “Metrics for assessing agents performance include: Navigation Error (NE) which is a measure of the distance between agent’s stopping location and the goal; Success Rate (SR) measures how often the agent stops within some acceptable distance of the goal; Success weighted by Path Length (SPL) [2] measu”
- Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, methodology): “…following evaluation metrics: Success Rate (SR), Success Rate Weighted by Path Length (SPL) (Anderson et al., 2018a), normalized Dynamic Time Warping (nDTW) (Magalhaes et al., 2019), success rate weighted by Dynamic Time Warping (Magalhaes et al., 2019) and Coverage weighted by Length Score (CLS).”
- Retouchdown: Adding Touchdown to StreetLearn as a Shareable Resource for Language Grounding Tasks in Street View (2020, background): “The same holds for DTW measures: Ilharco et al. (2019) report a success rate of 44% and corresponding SDTW of 38.3% for a fidelity-oriented version of the Re-inforced Cross-modal Matching agent (Wang et al., 2019).”
- Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (2020, methodology): “For full details on these metrics, see [2, 4, 18].”