VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation

By Yicong Hong, Qi Wu, Yuankai Qi, Cristian Rodriguez-Opazo, Stephen Gould. 2020

464 citations recorded by the source corpus.

Accuracy of many visiolinguistic tasks has benefited significantly from the application of vision-and-language (V&L) BERT. However, its application for the task of vision-and-language navigation (VLN) remains limited. One reason for this is the difficulty adapting the BERT architecture to the partially observable Markov decision process present in VLN, requiring history-dependent attention and decision making. In this paper we propose a recurrent BERT model that is time-aware for use in VLN. Specifically, we equip the BERT model with a recurrent function that maintains cross-modal state information for the agent. Through extensive experiments on R2R and REVERIE we demonstrate that our model can replace more complex encoder-decoder models to achieve state-of-the-art results. Moreover, our approach can be generalised to other transformer-based architectures, supports pre-training, and is capable of solving navigation and referring expression tasks simultaneously.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Multimodal Learning, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Multimodal Learning

In Robotics & Control

Citations in the corpus

Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, methodology): “Decoupled Multimodal Transformers enable extending transformer based multimodal embeddings and sequence-to-sequence prediction from domains with less than ten steps [35] to the hundreds of steps needed for the ALFRED benchmark [66].”
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (2021, background): “First, we need to develop more advanced component technologies integral to task learning, e.g., more advanced navigation modules through either more effective structures (Hong et al., 2020) or richer perceptions (Shen et al., 2019) to solve navigation bottleneck.”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, background): “Among all tasks, visual navigation has been most studied in the vision and robotics community [4, 13, 21, 38, 73, 87, 88].”
Playing Lottery Tickets with Vision and Language (2021, unknown): “This pretrain-thenfinetune learning paradigm has also been extended to diverse V+L tasks such as novel object captioning [26], visual dialog [51, 71], vision-language navigation [24, 49, 25], OCR-based VQA [73], robust VQA [38], and video-text pre-training [64, 63, 37, 82, 35].”
Language Grounding with 3D Objects (2021, unknown): “Models for these tasks extend Transformer representations to include action taking [31, 32] and invoke object classiﬁcation methods to create a semantic understanding of the world [33].”
Episodic Transformer for Vision-and-Language Navigation (2021, methodology): “Most relevant to our approach are VLN-BERT [42] and Recurrent VLBERT [28], which also employ transformers for VLN.”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “Recent work [11,16,21,23] employs BERT and its variants [7,20] to get better language understanding.”
ChatGPT for Robotics: Design Principles and Model Abilities (2023, background): “More recent works explore neural networks to implicitly keep track of the complex mapping between language and actions, but such techniques often require vast amounts of labeled data for training [9], [10], [29], [30]”
SQA3D: Situated Question Answering in 3D Scenes (2022, background): “Our task closely connects to the recent efforts on 3D language grounding (Dai et al., 2017; Chen et al., 2020; 2021; Hong et al., 2021b; Achlioptas et al., 2020; Wang et al., 2022; Azuma et al., 2022).”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, methodology): “Moreover, while HAMT noT2V is only 10% slower compared to [5], it still outperforms [5] in SR and SPL on val unseen split.”
Image Retrieval on Real-life Images with Pre-trained Vision-and-Language Models (2021, background): “This resembles the ﬁne-tuning of REF [23], as well as VLN-BERT [16].”
Multimodal Learning With Transformers: A Survey (2022, unknown): “vision-and-language navigation (VLN) [68], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83]), communication (e.”
VLP: A Survey on Vision-language Pre-training (2022, background): “PixelBERT[119] Image CNN-GFs Emb Single-stream No MLM+VLM COCO+VG VLR+NLVR+ VQA”