Speaker-Follower Models for Vision-and-Language Navigation

By Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Daniel Klein, Trevor Darrell. 2018

631 citations recorded by the source corpus.

Navigation guided by natural language instructions presents a challenging reasoning problem for instruction followers. Natural language instructions typically identify only a few high-level decisions and landmarks rather than complete low-level motor behaviors; much of the missing information must be inferred based on perceptual context. In machine learning settings, this is doubly challenging: it is difficult to collect enough annotated data to enable learning of this reasoning process from scratch, and also difficult to implement the reasoning process using generic sequence models. Here we describe an approach to vision-and-language navigation that addresses both these issues with an embedded speaker model. We use this speaker model to (1) synthesize new instructions for data augmentation and to (2) implement pragmatic reasoning, which evaluates how well candidate action sequences explain an instruction. Both steps are supported by a panoramic action space that reflects the granularity of human-generated instructions. Experiments show that all three components of this approach---speaker-driven data augmentation, pragmatic reasoning and panoramic action space---dramatically improve the performance of a baseline instruction follower, more than doubling the success rate over the best existing approach on a standard benchmark.

Subfields: Artificial Intelligence, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

Self-Instruct: Aligning Language Models with Self-Generated Instructions (2022, background): “Instructionfollowing models have also been of interest in the multi-modal learning literature (Fried et al., 2018; Shridhar et al., 2020; Min et al., 2022; Weir et al., 2022).”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, methodology): “Following the back-translation strategy that has been shown to be effective in Machine Translation (Sennrich et al., 2016) and Natural Language Navigation (Fried et al., 2018; Tan et al., 2019), we propose two methods to collect synthetic data.”
How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “Please refer to Fried et al. (2018) for the implementation details.”
On the Evaluation of Vision-and-Language Navigation Instructions (2021, methodology): “To establish benchmarks for navigation instruction generation, we evaluate existing English models (Fried et al., 2018; Tan et al., 2019) using human wayfinders.”
Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022, background): “…power of neural architectures, various works looked into creating instruction-following agents that can perform manipulation (Lynch & Sermanet, 2020; 2021), navigation (Fried et al., 2018; Wang et al., 2019; Majumdar et al., 2020), or both (Suglia et al., 2021; Hill et al., 2020; Fu et al., 2019).”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, methodology): “Existing beam-search [17, 47, 52] and backtracking solutions [24, 28] are infeasible due to the larger action and state spaces, long horizon, and inability to undo certain actions.”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “∗Corresponding author Equal Contribution Work performed during an internship at MSR (1)PRE-TRAINED VISION-AND-LANGUAGE BASED NAVIGATOR Most previous methods build on the sequence-tosequence architecture [26], where the instruction is encoded as a sequence of words, and the navigation trajectory is d”
SMArT: Training Shallow Memory-aware Transformers for Robotic Explainability (2019, background): “Recent advancements at the intersection of Computer Vision, Natural Language Processing and Robotics have tried to bring together the understanding of the visual world and that of natural language from the perspective of robots and embodied agents [1], [2], [3], [4], [5].”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, methodology): “Many papers report results trained on augmented data including instructions synthesized by a speaker model [8, 20, 21, 28].”
Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Text (2020, background): “Vision and language navigation (VLN) models learn to follow natural language directions that are longer that those considered in this work Misra et al. [2017, 2018], Anderson et al. [2018], Fried et al. [2018], Wang et al. [2019], Zhu et al. [2019].”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, methodology): “For our base instruction-to-navigation translation agent, we implement an encoder-decoder model similar to Fried et al. (2018).”
Embodied BERT: A Transformer Model for Embodied, Language-guided Visual Task Completion (2021, methodology): “For language supervision, we could train and apply a speaker model for ALFRED to generate additional training data for new expert demonstrations, providing an initial multimodal alignment for EmBERT, a strategy shown effective in VLN tasks (Fried et al., 2018).”
Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Grounding (2020, methodology): “7) is trained without data augmentation from model-generated instructions (Fried et al., 2018; Tan et al., 2019) and with hyperparameters tuned for RxR.”
Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, methodology): “Following the Speaker-Follower model (Fried et al., 2018), at time step t , we provide the agent with a feature set I cur t representing the current panoramic view.”
Language as an Abstraction for Hierarchical Deep Reinforcement Learning (2019, background): “[19] Daniel Fried, Ronghang Hu, Volkan Cirik, Anna Rohrbach, Jacob Andreas, Louis-Philippe Morency, Taylor Berg-Kirkpatrick, Kate Saenko, Dan Klein, and Trevor Darrell.”
Robust Navigation with Language Pretraining and Stochastic Sampling (2019, methodology): “Following (Fried et al., 2018), we use a panoramic action space and the ResNet image features provided by (Anderson et al., 2018).”
Self-Monitoring Navigation Agent via Auxiliary Progress Estimation (2019, neutral): “Interestingly, the performance improvement introduced by data augmentation is smaller than from Speaker-Follower on the validation sets (see Table 1 for comparison).”
Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoring (2021, methodology): “Many models have been developed, such as the Speaker-Follower model (Fried et al., 2018), the Self-Monitoring Navigation Agent(Ma et al.”