IQA: Visual Question Answering in Interactive Environments

By Daniel Gordon, Aniruddha Kembhavi, Mohammad Rastegari, Joseph Redmon, Dieter Fox, Ali Farhadi. 2017

435 citations recorded by the source corpus.

We introduce Interactive Question Answering (IQA), the task of answering questions that require an autonomous agent to interact with a dynamic visual environment. IQA presents the agent with a scene and a question, like: "Are there any apples in the fridge?" The agent must navigate around the scene, acquire visual understanding of scene elements, interact with objects (e.g. open refrigerators) and plan for a series of actions conditioned on the question. Popular reinforcement learning approaches with a single controller perform poorly on IQA owing to the large and diverse state space. We propose the Hierarchical Interactive Memory Network (HIMN), consisting of a factorized set of controllers, allowing the system to operate at multiple levels of temporal abstraction. To evaluate HIMN, we introduce IQUAD V1, a new dataset built upon AI2-THOR [35], a simulated photo-realistic environment of configurable indoor scenes with interactive objects. IQUAD V1 has 75,000 questions, each paired with a unique scene configuration. Our experiments show that our proposed model outperforms popular single controller based methods on IQUAD V1. For sample questions and results, please view our video: https://youtu.be/pXd3C-1jr98.

Subfields: Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Experience Grounds Language (2020, background): “Understanding this social layer is not only necessary, but will make clear complexities around implicature and commonsense that so obviously arise in corpora, but with such sparsity that they cannot properly be learned (Gordon et al., 2018).”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “Embodied Language Learning : To address language learning in embodied domains, a number of interactive environments have been proposed: BabyAI (Chevalier-Boisvert et al., 2019), Room2Room (Anderson et al., 2018b), ALFRED (Shridhar et al., 2020), InteractiveQA (Gordon et al., 2018), EmbodiedQA (Das e”
ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Existing datasets for visual question answering in embodied environments use templated language or static scenes [15, 20, 54, 56].”
OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledge (2019, background): “e question. Modular networks [2,18,23] leverage the compositional nature of the language in deep neural networks. These approaches have been extended to the video domain as well [20,34,42]. Recently, [15,10] address the problem of question answering in an interactive environment. None of these appro”
Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “Indeed, even agents operating in 3D simulators that notionally support continuous motion typically use discretized action spaces in practice [62, 16, 18, 47].”
Grad-CAM: Visual Explanations from Deep Networks via Gradient-Based Localization (2016, neutral): “2017b) and embodied question answering (Das et al. 2018; Gordon et al. 2017).”
Embodied Question Answering (2017, background): “[32] studied the task of EmbodiedQA in the AI2-THOR environment [33].”
Continuous Scene Representations for Embodied AI (2022, background): “Recently, various embodied tasks such as navigation [5,8,37,54,56,63], instruction following [1,47], embodied question answering [15, 23], object manipulation [18,21,22,46], and room rearrangement [4,52] have been of interest.”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “IQA [106] is another work that focuses on tackling the”
YOLOv3: An Incremental Improvement (2018, background): “We have a camera-ready deadline [4] and we need to cite some of the random updates I made to YOLO but we don’t have a source.”
AI2-THOR: An Interactive 3D Environment for Visual AI (2017, methodology): “[10] perform interactive visual question answering using AI2-THOR environment.”
Cognitive Mapping and Planning for Visual Navigation (2017, methodology): “[10, 27, 45, 61] use similar 2D spatial representations, Mirowski et al.”
A Survey of Reinforcement Learning Informed by Natural Language (2019, background): “Embodied Q&A (EQA) extends this setting, by requiring agent to explore and navigate the environment in order to answer queries [Das et al. , 2018a; Gordon et al. , 2018 ] , for example, “How many mugs are in the kitchen?” or “Is there a tomato in the fridge?”.”
Vision-and-Dialog Navigation (2019, background): “Dataset —Language Context— —Visual Context— Human Amb UnderS Temporal Real-world Temporal Shared MARCO[8, 9], DRIF[10] 3 7 7 1I 7 Dynamic R2R[7], Touchdown[11] 3 7 7 1I 3 Dynamic EQA[12], IQA[13] 7 7 3 1Q 7 Dynamic CLEVR[14] 7 7 1Q 7 Static VQA[15, 16, 17] 3 7 1Q 3 Static CLEVR-Dialog[18] 7 7 2D 7 S”
Skill Induction and Planning with Latent Language (2021, background): “, 2017), embodied question answering (Das et al., 2018; Gordon et al., 2018) and dialog tasks that combine the two (Tellex et al.”
Learning to Navigate Unseen Environments: Back Translation with Environmental Dropout (2019, background): “…including instruction-based navigation (Chaplot et al., 2018; Anderson et al., 2018b), target-driven navigation (Zhu et al., 2017b; Gupta et al., 2017), embodied question answering (Das et al., 2018), interactive question answering (Gordon et al., 2018), and task planning (Zhu et al., 2017a).”
TOUCHDOWN: Natural Language Navigation and Spatial Reasoning in Visual Street Environments (2018, methodology): “R2R uses panorama graphs of house environments for the task of navigation instruction following.”
Neural Modular Control for Embodied Question Answering (2018, background): “[30] in the AI2-THOR environment [37].”