Episodic Transformer for Vision-and-Language Navigation

By Alexander Pashevich, Cordelia Schmid, Chen Sun. 2021

233 citations recorded by the source corpus.

Interaction and navigation defined by natural language instructions in dynamic environments pose significant challenges for neural agents. This paper focuses on addressing two challenges: handling long sequence of subtasks, and understanding complex human instructions. We propose Episodic Transformer (E.T.), a multimodal transformer that encodes language inputs and the full episode history of visual observations and actions. To improve training, we leverage synthetic instructions as an intermediate representation that decouples understanding the visual appearance of an environment from the variations of natural language instructions. We demonstrate that encoding the history with a transformer is critical to solve compositional tasks, and that pretraining and joint training with synthetic instructions further improve the performance. Our approach sets a new state of the art on the challenging ALFRED benchmark, achieving 38.4% and 8.5% task success rates on seen and unseen test splits.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Do As I Can, Not As I Say: Grounding Language in Robotic Affordances (2022, background): “Recent works have studied how to ground modern language models, by training them to accept additional environment inputs [30, 31, 32, 33, 34] or to directly output actions [35, 36, 37].”
Skill Induction and Planning with Latent Language (2021, background): “…seq2seq2seq S+(Shridhar et al., 2020) MOCA(Singh et al., 2020) Modular (Corona et al., 2021) ABP (Kim et al., 2021) EmBERT (Suglia et al., 2021) ET (Pashevich et al., 2021) HLSM(Blukis et al., 2021) * * HiTUT (Zhang and Chai, 2021) FILM (Min et al., 2021) Table 3: Detailed comparison of…”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, methodology): “EmBERT achieves competitive performance with state of the art models on the ALFRED leaderboard test sets (Table 2), surpassing all but ET (Pashevich et al., 2021) and ABP (Kim et al., 2021) on Seen test fold performance (Table 3) at the time of writing.”
FILM: Following Instructions in Language with Modular Methods (2021, methodology): “Baselines There are two kinds of baselines: those that use low-level sequential instructions (Kim et al., 2021; Zhang & Chai, 2021; Nguyen et al., 2021; Pashevich et al., 2021) and those that do not (Nottingham et al.”
RT-1: Robotics Transformer for Real-World Control at Scale (2022, methodology): “As in RT-1, several works use language commands processed with Transformers as a robust framework for specifying and generalizing to new tasks (Zhang & Chai, 2021; Pashevich et al., 2021; Silva et al., 2021; Jang et al., 2021; Ahn et al., 2022; Nair et al., 2022).”
History Aware Multimodal Transformer for Vision-and-Language Navigation (2021, background): “Differently from [37], we propose a hierarchical encoding of the panoramic observation history and optimize the whole model in end-to-end training.”
Multimodal Learning With Transformers: A Survey (2022, unknown): “vision-and-language navigation (VLN) [68], [74], [75], [76], [77], [78], [79], [80], [81], [82], [83]), communication (e.”
SQA3D: Situated Question Answering in 3D Scenes (2022, background): “…usually comprise multi-modal fusion (Tenenbaum & Freeman, 1996; Perez et al., 2018) and are trained from scratch (Wang et al., 2018; Fried et al., 2018; Wang et al., 2019), while recent efforts would employ pre-trained models (Pashevich et al., 2021; Hong et al., 2021a; Suglia et al., 2021).”
NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models (2023, background): “…et al. 2022; Li, Tan, and Bansal 2022; Tan, Yu, and Bansal 2019; Parvaneh et al. 2020; Li and Bansal 2023), memory mechanism (Chen et al. 2021b; Pashevich, Schmid, and Sun 2021; Hong et al. 2023), pre-training (Hao et al. 2020; Chen et al. 2022a; Qiao et al. 2023a; Wang et al. 2023) have been…”
Online Continual Learning For Interactive Instruction Following Agents (2024, methodology): “Most previous methods (Singh et al., 2021; Pashevich et al., 2021; Min et al., 2022; Kim et al., 2023) for object localization utilize a two-stage approach, separating it into object class prediction and mask generation to enhance object localization.”
Instruction-driven history-aware policies for robotic manipulations (2022, methodology): “Recent works in Vision-and-Language Navigation (VLN) [74, 75, 76] further demonstrate that the Trans-former allows to better leverage previous observations to improve multi-modal action prediction.”
Attention Bottlenecks for Multimodal Fusion (2021, methodology): “[51] Alexander Pashevich, Cordelia Schmid, and Chen Sun.”
LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2022, methodology): “In navigation-only VLN datasets such as R2R [2], models that generate the action sequence end-to-end with a Transformer model can already achieve a good performance [25,35].”