Embodied Question Answering

By Abhishek Das, Samyak Datta, Georgia Gkioxari, Stefan Lee, Devi Parikh, Dhruv Batra. 2017

794 citations recorded by the source corpus.

We present a new AI task - Embodied Question Answering (EmbodiedQA) - where an agent is spawned at a random location in a 3D environment and asked a question ('What color is the car?'). In order to answer, the agent must first intelligently navigate to explore the environment, gather necessary visual information through first-person (egocentric) vision, and then answer the question ('orange'). EmbodiedQA requires a range of AI skills - language understanding, visual recognition, active perception, goal-driven navigation, commonsense reasoning, long-term memory, and grounding language into actions. In this work, we develop a dataset of questions and answers in House3D environments [1], evaluation metrics, and a hierarchical model trained with imitation and reinforcement learning.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Multimodal Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Multimodal Learning

Citations in the corpus

Experience Grounds Language (2020, background): “…descendant from image captioning (Farhadi et al., 2010; Mitchell et al., 2012), a myriad of tasks on visual question answering (Antol et al., 2015; Das et al., 2018), natural language and visual reasoning (Suhr et al., 2019b), visual commonsense (Zellers et al., 2019a), and multilingual…”
Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions (2020, background): “Embodied agents that use language are also being explored (Das et al., 2018; Savva et al., 2019; Szlam et al., 2019; Urbanek et al., 2019).”
ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks (2019, background): “3.2, we do miss some families of tasks including visually grounded dialog [4, 45], embodied tasks like question answering [7] and instruction following [8], and text generation tasks like image and video captioning [5].”
Imitating Interactive Intelligence (2020, background): “, 2017) or question-answering (Das et al., 2018).”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “Embodied Language Learning : To address language learning in embodied domains, a number of interactive environments have been proposed: BabyAI (Chevalier-Boisvert et al., 2019), Room2Room (Anderson et al., 2018b), ALFRED (Shridhar et al., 2020), InteractiveQA (Gordon et al., 2018), EmbodiedQA (Das e”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Existing datasets for visual question answering in embodied environments use templated language or static scenes [15, 20, 54, 56].”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “e question. Modular networks [2,18,23] leverage the compositional nature of the language in deep neural networks. These approaches have been extended to the video domain as well [20,34,42]. Recently, [15,10] address the problem of question answering in an interactive environment. None of these appro”
Towards Learning a Generic Agent for Vision-and-Language Navigation via Pre-Training (2020, background): “Learning to navigate in a photorealistic home environment based on natural language instructions has attracted increasing research interest [23, 14, 7, 3, 6], as it provides insight into core scientific questions about multimodal representations.”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “Indeed, even agents operating in 3D simulators that notionally support continuous motion typically use discretized action spaces in practice [62, 16, 18, 47].”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, background): “1, we do miss some families of tasks including image and video captioning [7], visual dialog [12], embodied question answering [11] and instruction following [3].”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, background): “object detection [21], semantic segmentation [37] to image captioning [55,7,18,29], visual question answering [3,20,42,46] and more recently, visual dialog [11, 13,12] and embodied question answering [10,23]. While 1 http://gradcam.cloudcv.org arXiv:1610.02391v4 [cs.CV] 3 Dec 2019 2 Ramprasaath R. S”
Continuous Scene Representations for Embodied AI (2022, background): “Recently, various embodied tasks such as navigation [5,8,37,54,56,63], instruction following [1,47], embodied question answering [15, 23], object manipulation [18,21,22,46], and room rearrangement [4,52] have been of interest.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “3) Datasets: The EQA [61] dataset is based on House3D, a subset of the popular SUNCG [33] dataset with synthesized rooms and layouts that is similar to the Replica dataset [107].”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “…have also gone beyond specifying goals as a desired location in space and finding objects of interest as done in this paper, for example, Wu et al. (2018) generalize the goal specification to also include rooms of interest, and Das et al. (2018) allow goal specification via templated questions.”
Speaker-Follower Models for Vision-and-Language Navigation (2018, background): “While a possible way to address this challenge is to learn a hierarchical policy such as in [13], in our work we directly allow the agent to reason about high-level actions, using a panoramic action space with panoramic representation, converted with built-in mapping from low-level visuomotor contro”
Building Generalizable Agents with a Realistic and Rich 3D Environment (2018, methodology): “On the contrary, we focus on building a platform that intersects with multiple research directions, such as object and scene understanding, 3D navigation, embodied question answering (Das et al., 2017a), while allowing users to customize the level of complexity to their needs.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, methodology): “While originally developed for visual question answering, [ Das et al. , 2018b ] and [ Bahdanau et al. , 2019 ] successfully apply variants of this idea to RL tasks.”
Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigation (2018, background): “Recently, vision-language grounded embodied agents have received increased attention [32, 22, 7] due to their popularity in many intriguing real-world applications, e.”
MINOS: Multimodal Indoor Simulator for Navigation in Complex Environments (2017, methodology): “Since then, a number of independent efforts have investigated navigation in indoor environments using the SUNCG dataset (Wu et al., 2017; Brodeur et al., 2017; Das et al., 2017) and the Matterport3D dataset (Anderson et al., 2017).”