Chasing Ghosts: Instruction Following as Bayesian State Tracking

By Peter Anderson, Ayush Shrivastava, Devi Parikh, Dhruv Batra, Stefan Lee. 2019

79 citations recorded by the source corpus.

A visually-grounded navigation instruction can be interpreted as a sequence of expected observations and actions an agent following the correct trajectory would encounter and perform. Based on this intuition, we formulate the problem of finding the goal location in Vision-and-Language Navigation (VLN) within the framework of Bayesian state tracking - learning observation and motion models conditioned on these expectable events. Together with a mapper that constructs a semantic spatial map on-the-fly during navigation, we formulate an end-to-end differentiable Bayes filter and train it to identify the goal by predicting the most likely trajectory through the map according to the instructions. The resulting navigation policy constitutes a new approach to instruction following that explicitly models a probability distribution over states, encoding strong geometric and algorithmic priors while enabling greater explainability. Our experiments show that our approach outperforms a strong LingUNet baseline when predicting the goal location on the map. On the full VLN task, i.e. navigating to the goal location, our approach achieves promising results with less reliance on navigation constraints.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Probabilistic & Bayesian Methods

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Probabilistic & Bayesian Methods

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, neutral): “…al., 2018) 25 23 S-Follower (Fried et al., 2018) 35 28 RCM (Wang et al., 2019) 43 38 SMNA (Ma et al., 2019a) 48 35 Regretful (Ma et al., 2019b) 48 40 FAST-Short (Ke et al., 2019) 54 41 EnvDrop (Tan et al., 2019) 51 47 PRESS (Li et al., 2019b) 49 45 ALTR (Huang et al., 2019a) 48 45 CG (Anderson et”
Improving Vision-and-Language Navigation with Image-Text Pairs from the Web (2020, background): “In this setting, a high-probability path is typically generated by performing beam search through the environment and ranking paths according to either: (1) their probability under a ‘follower’ model [3,8,13,20,21,30,31], as in Figure 1 (left), or (2) by how well they explain the instruction accordi”
Language-Conditioned Imitation Learning for Robot Manipulation Tasks (2020, background): “Neural approaches scale imitation learning [27, 4, 20, 1, 9] to high-dimensional spaces by enabling agents to learn task-speciﬁc feature representations.”
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (2020, background): “Others use precise agent localization to construct spatial maps of the environment on which to reason about paths [3].”
Few-shot Object Grounding and Mapping for Natural Language Robot Instruction Following (2020, methodology): “We use the Position Visitation Network [26, 6], but our approach is applicable to other policy architectures [42, 43].”
A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution (2021, background): “Similar semantic 2D maps have been successfully used in navigation [7, 8, 48, 49] and more recently even in mobile manipulation instruction-following tasks [23].”
Topological Planning with Transformers for Vision-and-Language Navigation (2020, methodology): “For example, [2] use metric maps for VLN, and [17] use transformers for explorative tasks by storing observations and poses as memory.”
Learning to Map Natural Language Instructions to Physical Quadcopter Control using Simulated Flight (2019, background): “This approach is related to neural network models that construct maps [33, 34, 35, 36] or perform planning [37].”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Embodied navigation also aids AI agents to develop behavior such as instruction following [38,4,82,95,3], city navigation [18,64,63,94], question answering [21,22,34,97,24], and active visual recognition [105,104].”