Shifting the Baseline: Single Modality Performance on Visual Navigation & QA

By Jesse Thomason, Daniel Gordon, Yonatan Bisk. 2018

85 citations recorded by the source corpus.

We demonstrate the surprising strength of unimodal baselines in multimodal domains, and make concrete recommendations for best practices in future research. Where existing work often compares against random or majority class baselines, we argue that unimodal approaches better capture and reflect dataset biases and therefore provide an important comparison when assessing the performance of multimodal techniques. We present unimodal ablations on three recent datasets in visual navigation and QA, seeing an up to 29% absolute gain in performance over published baselines.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “Careful design is key, as visually augmented tasks can fail to require sensory perception (Thomason et al., 2019a).”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Previous work established that learned agents without visual inputs, language inputs, or both performed better than random agents and were competitive with initial baselines for several navigation and question answering tasks [49].”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, background): “Successful navigation largely depends on precise joint understanding of natural language instructions and the visual states [29].”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Concurrent to our work, [42, 24, 25, 26] studies the VLN tasks from various aspects, and [30] introduces a variant of the VLN task to find objects by requesting language assistance when needed.”
Vision-and-Dialog Navigation (2019, background): “Random baselines can be outperformed by unimodal model ablations—agents that consider only visual input, only language input, or neither—on VLN tasks [35].”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, background): “A number of VLN datasets situated in photorealistic 3D reconstructions of real locations contain human instructions or dialogue: R2R (Anderson et al., 2018b), Touchdown (Chen et al., 2019; Mehta et al., 2020), CVDN (Thomason et al., 2019b) and REVERIE (Qi et al., 2020).”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “Does understanding language improve generalizability? Our agent is assisted with both language and visual instructions; similar to Thomason et al. (2019a), we disentangle the useful-”
Robust Navigation with Language Pretraining and Stochastic Sampling (2019, methodology): “• S EQ 2S EQ : sequence-to-sequence model proposed by Anderson et al. as a baseline for the R2R benchmark (Anderson et al., 2018) and analyzed in (Thomason et al., 2019).”
What Makes Training Multi-Modal Classification Networks Hard? (2019, background): “Anecdotally, the performance drop with multiple input streams appears to be common and was noted in [24, 3, 38, 44].”
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (2020, methodology): “We perform a number of input-modality ablations to assess the biases and baselines in this new setting (including models without perception or instructions as suggested in [27]).”
Audio-Visual Embodied Navigation (2019, background): “Following these ﬁndings, recent visual AI agents aggregate egocentric visual inputs [118, 117, 65, 90], often with a spatio-temporal memory [41, 81, 46, 105, 98].”
What Should I Do Now? Marrying Reinforcement Learning and Symbolic Planning (2019, methodology): “For the IQA experiments we use the “Language Only” baselines presented in [31] as b.”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “Recent AI agents also aggregate egocentric visual inputs [112,111,63,86,46], often with a spatio-temporal memory [40,79,44,101].”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “based learning [99,103], meta-learning [98], unimodal baselines [90], 3D point clouds [97], and effective exploration [95,78,16,74] to improve embodied navigational agents.”
Embodied Question Answering in Photorealistic Environments With Point Cloud Perception (2019, background): “[42]), PC, RGB, and PC+RGB, with the 2 different language variations, None and Question.”
Misleading Failures of Partial-input Baselines (2019, background): “Partial-input baselines have identiﬁed artifacts in many datasets, e.g., SNLI (Gururangan et al., 2018; Poliak et al., 2018), VQA (Goyal et al., 2017), EmbodiedQA (Anand et al., 2018), visual dialogue (Massiceti et al., 2018), and visual navigation (Thomason et al., 2019).”
RUBi: Reducing Unimodal Biases in Visual Question Answering (2019, background): “In the language and vision context, some works evaluate unimodal baselines [37, 38] or leverage language priors [39].”
Stay on the Path: Instruction Fidelity in Vision-and-Language Navigation (2019, methodology): “Thomason et al. (2019) presented an analysis on R2R dataset, where the trained agent continued to perform surprisingly well in the absence of language inputs.”