Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning

By Khanh Nguyen, Hal Daumé Iii. 2019

172 citations recorded by the source corpus.

Mobile agents that can leverage help from humans can potentially accomplish more complex tasks than they could entirely on their own. We develop “Help, Anna!” (HANNA), an interactive photo-realistic simulator in which an agent fulfills object-finding tasks by requesting and interpreting natural language-and-vision assistance. An agent solving tasks in a HANNA environment can leverage simulated human assistants, called ANNA (Automatic Natural Navigation Assistants), which, upon request, provide natural language and visual instructions to direct the agent towards the goals. To address the HANNA problem, we develop a memory-augmented neural agent that hierarchically models multiple levels of decision-making, and an imitation learning algorithm that teaches the agent to avoid repeating past mistakes while simultaneously predicting its own chances of making future progress. Empirically, our approach is able to ask for help more effectively than competitive baselines and, thus, attains higher task success rate on both previously seen and previously unseen environments.

Subfields: Computer Vision and Pattern Recognition, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

How Much Can CLIP Benefit Vision-and-Language Tasks? (2021, methodology): “We use the standard automatic evaluation metrics including CIDEr (Anderson et al., 2016), BLEU (Papineni et al., 2002), METEOR (Lavie and Agarwal, 2007), and SPICE (Anderson et al., 2016).”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, methodology): “(HANNA) [22].”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, background): “These EQA tasks have recently been extended in the form of conversations, where agents compensate by querying oracles for insufficient information to perform the task [157].”
Robots That Ask For Help: Uncertainty Alignment for Large Language Model Planners (2023, background): “Since Simple Set does not provide calibrated coverage, we first find ϵ =0 .”
Interactive Learning from Activity Description (2021, background): “The goal of a request-fulfilling agent is to map a given language request in a situated environment to an execution that accomplishes the intent of the request [2, 5, 9, 10, 17, 44, 48, 51, 58, 61].”
Cognitive Architectures for Language Agents (2023, background): “Agents capable of generating language may ask for help (Ren et al., 2023; Nguyen et al., 2022b; 2019; Nguyen and Daumé III, 2019) or clarification (Biyik and Palan, 2019; Sadigh et al., 2017; Padmakumar et al., 2022; Thomason et al., 2020; Narayan-Chen et al., 2019) – or entertain or emotionally…”
Language and Visual Entity Relationship Graph for Agent Navigation (2020, background): “A great variety of scenarios have been set up for relevant research, such as long range indoor and street view navigation with comprehensive instructions [3, 5, 17], communication based visual navigation [7, 25, 34], and navigation for object localization and visual question answering [6, 29].”
Improving Cross-Modal Alignment in Vision Language Navigation via Syntactic Information (2021, background): “Vision-Language Navigation deﬁnes the task of requiring an agent to navigate through a visual environment based on natural language instructions (Anderson et al., 2018b; Misra et al., 2018; Chen et al., 2019; Jain et al., 2019; Nguyen and Daumé III, 2019; Thomason et al., 2020).”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, methodology): “Following these work, dialog-based navigation such as CVDN [52], VNLA [41] and HANNA [40], navigation for localising a remote object such as REVERIE [44], VLN in continuous environment [22], and multilingual navigation with spatialtemporal grounding such as RxR [24] have been proposed for further re”