Counterfactual Vision-and-Language Navigation via Adversarial Path Sampling

By Tsu-Jui Fu, X. Wang, Matthew F. Peterson, Scott T. Grafton, Miguel P. Eckstein, William Yang Wang. 2019

47 citations recorded by the source corpus.

Vision-and-Language Navigation (VLN) is a task where agents must decide how to move through a 3D environment to reach a goal by grounding natural language instructions to the visual surroundings. One of the problems of the VLN task is data scarcity since it is difficult to collect enough navigation paths with human-annotated instructions for interactive environments. In this paper, we explore the use of counterfactual thinking as a human-inspired data augmentation method that results in robust models. Counterfactual thinking is a concept that describes the human propensity to create possible alternatives to life events that have already occurred. We propose an adversarial-driven counterfactual reasoning model that can consider effective conditions instead of low-quality augmented data. In particular, we present a model-agnostic adversarial path sampler (APS) that learns to sample challenging paths that force the navigator to improve based on the navigation performance. APS also serves to do pre-exploration of unseen environments to strengthen the model's ability to generalize. We evaluate the influence of APS on the performance of different VLN baseline models using the room-to-room dataset (R2R). The results show that the adversarial training process with our proposed APS benefits VLN models under both seen and unseen environments. And the pre-exploration process can further gain additional improvements under unseen environments.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

On the Evaluation of Vision-and-Language Navigation Instructions (2021, background): “Most recent efforts (e.g., Fu et al., 2019; Huang et al., 2019; Jain et al., 2019; Wang et al., 2019, etc.) have used the Room-to-Room (R2R) dataset (Anderson et al., 2018b), which contains 4675 unique paths in the train split, 340 in the valseen split (same environments, new paths), and an…”
Room-and-Object Aware Knowledge Reasoning for Remote Embodied Referring Expression (2021, methodology): “Numerous methods [47, 48, 19, 7, 12, 10, 46, 36, 17, 20] have been proposed to address the VLN task.”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “FAST [19] resorts self-correction navigation, while APS [12] samples adversarial paths for training to enhance the model’s ability to generalise.”