Improving Vision-and-Language Navigation with Image-Text Pairs from the Web

By Arjun Majumdar, Ayush Shrivastava, Stefan Lee, Peter Anderson, Devi Parikh, Dhruv Batra. 2020

277 citations recorded by the source corpus.

Following a navigation instruction such as 'Walk down the stairs and stop at the brown sofa' requires embodied AI agents to ground scene elements referenced via language (e.g. 'stairs') to visual content in the environment (pixels corresponding to 'stairs'). We ask the following question -- can we leverage abundant 'disembodied' web-scraped vision-and-language corpora (e.g. Conceptual Captions) to learn visual groundings (what do 'stairs' look like?) that improve performance on a relatively data-starved embodied perception task (Vision-and-Language Navigation)? Specifically, we develop VLN-BERT, a visiolinguistic transformer-based model for scoring the compatibility between an instruction ('...stop at the brown sofa') and a sequence of panoramic RGB images captured by the agent. We demonstrate that pretraining VLN-BERT on image-text pairs from the web before fine-tuning on embodied path-instruction data significantly improves performance on VLN -- outperforming the prior state-of-the-art in the fully-observed setting by 4 absolute percentage points on success rate. Ablations of our pretraining curriculum show each stage to be impactful -- with their combination resulting in further positive synergistic effects.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

Citations in the corpus

Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “…pre-training has brought leaping advances in vision+language understanding tasks such as VQA and VCR, with great potential in extending to visual captioning [84, 70], visual dialog [47, 68], vision-language navigation [18, 43], as well as video-and-language representation learning [62, 61, 41, 31].”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “Compared to the trajectory re-ranking compatibility model proposed by Majumdar et al. (2020), we use a dual encoder architecture rather than dense cross-attention.”
Pre-Trained Language Models for Interactive Decision-Making (2022, background): “This includes tasks that are not purely linguistic, and require nontrivial planning and reasoning capabilities [24, 13]: for example, instruction following, vision-language navigation, and visual question answering.”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “At the same time, by leveraging representation power of neural architectures, various works looked into creating instruction-following agents that can perform manipulation (Lynch & Sermanet, 2020; 2021), navigation (Fried et al., 2018; Wang et al., 2019; Majumdar et al., 2020), or both (Suglia et al”
Transformers in Vision: A Survey (2021, methodology): “Self-attention based Transformer networks were used earlier in [147], [148] for the visual and language navigation (VLN).”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, methodology): “…incorporate multimodal transformer models that require large memory from recurrence (Hong et al., 2020), beam search over potential action sequences (Majumdar et al., 2020), or shallow layers without largescale pretraining to encode long histories (Pashevich et al., 2021; Magassouba et al., 2021).”
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO (2020, background): “, 2017), Vision and Language Navigation (Majumdar et al., 2020), Referring Expressions (Yu et al.”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “In terms of VLN pre-training, PRESS fine tunes a pre-trained language BERT to encode instructions [29], PREVALENT trains a V&L BERT on a large amount of image-text-action triplets from scratch to learn navigation-oriented textual representations [14], and VLN-BERT [37] fine-tunes a ViLBERT [33] on i”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “Recent work [11,16,21,23] employs BERT and its variants [7,20] to get better language understanding.”
Playing Lottery Tickets with Vision and Language (2021, unknown): “This pretrain-thenfinetune learning paradigm has also been extended to diverse V+L tasks such as novel object captioning [26], visual dialog [51, 71], vision-language navigation [24, 49, 25], OCR-based VQA [73], robust VQA [38], and video-text pre-training [64, 63, 37, 82, 35].”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, methodology): “Given the input pair ( W , H T ) where T is the length of full trajectory, we can apply common proxy tasks as in vision-and-language pretraining [40, 44], including Masked Language Modeling (MLM), Masked Region Modeling (MRM) and Instruction Trajectory Matching (ITM).”
Foundation models in robotics: Applications, challenges, and the future (2023, unknown): “ViNT encodes visual observations and visual goals using an EfﬁcientNet CNN and predicts temporal distance and normalized actions in an embodiment-agnostic manner.”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “Other works improve vision and language representations (Hu et al., 2019; Li et al., 2019; Huang et al., 2019b,a; Hao et al., 2020; Majumdar et al., 2020) and propose an additional progress monitor module (Ma et al., 2019b,a; Ke et al., 2019) and object and action aware modules (Qi et al., 2020b)…”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, methodology): “Data augmentation [36, 58, 32, 53, 41, 15, 56], memory mechanism [6, 57, 42], pre-training [39, 21, 20, 65, 44] have been adopted to alleviate data scarcity.”
Language Grounding with 3D Objects (2021, background): “Models for these tasks extend Transformer representations to include action taking [31, 32] and invoke object classification methods to create a semantic understanding of the world [33].”
Episodic Transformer for Vision-and-Language Navigation (2021, methodology): “Most relevant to our approach are VLN-BERT [42] and Recurrent VLBERT [28], which also employ transformers for VLN.”