History Aware Multimodal Transformer for Vision-and-Language Navigation

By Shizhe Chen, Pierre-Louis Guhur, Cordelia Schmid, Ivan Laptev. Neural Information Processing Systems, 2021

388 citations recorded by the source corpus.

Vision-and-language navigation (VLN) aims to build autonomous visual agents that follow instructions and navigate in real scenes. To remember previously visited locations and actions taken, most approaches to VLN implement memory using recurrent states. Instead, we introduce a History Aware Multimodal Transformer (HAMT) to incorporate a long-horizon history into multimodal decision making. HAMT efficiently encodes all the past panoramic observations via a hierarchical vision transformer (ViT), which first encodes individual images with ViT, then models spatial relation between images in a panoramic observation and finally takes into account temporal relation between panoramas in the history. It, then, jointly combines text, history and current observation to predict the next action. We first train HAMT end-to-end using several proxy tasks including single step action prediction and spatial relation prediction, and then use reinforcement learning to further improve the navigation policy. HAMT achieves new state of the art on a broad range of VLN tasks, including VLN with fine-grained instructions (R2R, RxR), high-level instructions (R2R-Last, REVERIE), dialogs (CVDN) as well as long-horizon VLN (R4R, R2R-Back). We demonstrate HAMT to be particularly effective for navigation tasks with longer trajectories.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/6f681faaa985ed38bc9b30777d57d9e1e3765861

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yu Qiao
2. Luc Van Gool
3. Cordelia Schmid
4. Andrew Zisserman
5. Yi Yang
6. Dacheng Tao
7. Jitendra Malik
8. Limin Wang

Citations in the corpus

• Instruction-driven history-aware policies for robotic manipulations (2022, background): “Recent works in Vision-and-Language Navigation (VLN) [74, 75, 76] further demonstrate that the Trans-former allows to better leverage previous observations to improve multi-modal action prediction.”
• From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, methodology): “The task has been extensively studied in [Wang et al., 2018; Fried et al., 2018; Tan et al., 2019; Huang et al., 2019; Wang et al., 2019; Hong et al., 2021; Chen et al., 2021].”
• Multimodal Learning With Transformers: A Survey (2022, neutral): “vision-and-language navigation (VLN) [68], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83]), communication (e.”
• InternVideo: General Video Foundation Models via Generative and Discriminative Learning (2022, methodology): “The history-enhanced planner is a customized variant of HAMT [101] which uses a concatenation of depth embedding and RGB embedding as the input embedding.”
• NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, methodology): “Data augmentation [36, 58, 32, 53, 41, 15, 56], memory mechanism [6, 57, 42], pre-training [39, 21, 20, 65, 44] have been adopted to alleviate data scarcity.”