Multi-Target Embodied Question Answering

By Licheng Yu, Xinlei Chen, Georgia Gkioxari, Mohit Bansal, Tamara L. Berg, Dhruv Batra. 2019

135 citations recorded by the source corpus.

Embodied Question Answering (EQA) is a relatively new task where an agent is asked to answer questions about its environment from egocentric perception. EQA as introduced in [8] makes the fundamental assumption that every question, e.g., ``what color is the car?", has exactly one target (``car") being inquired about. This assumption puts a direct limitation on the abilities of the agent. We present a generalization of EQA -- Multi-Target EQA (MT-EQA). Specifically, we study questions that have multiple targets in them, such as ``Is the dresser in the bedroom bigger than the oven in the kitchen?", where the agent has to navigate to multiple locations (``dresser in bedroom", ``oven in kitchen") and perform comparative reasoning (``dresser" bigger than ``oven") before it can answer a question. Such questions require the development of entirely new modules or components in the agent. To address this, we propose a modular architecture composed of a program generator, a controller, a navigator, and a VQA module. The program generator converts the given question into sequential executable sub-programs; the navigator guides the agent to multiple locations pertinent to the navigation-related sub-programs; and the controller learns to select relevant observations along its path. These observations are then fed to the VQA module to predict the answer. We perform detailed analysis for each of the model components and show that our joint model can outperform previous methods and strong baselines by a significant margin.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks (2019, background): “Existing datasets for visual question answering in embodied environments use templated language or static scenes [15, 20, 54, 56].”
A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “[104] and [105] further improved the PACMAN model by the Neural Modular Control (NMC) where the higher-level master policy proposes seman-”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, unknown): “For reference, human evaluators correlated with each other in ⇢ 2 [0 .”
From Seeing to Moving: A Survey on Learning for Visual Indoor Navigation (VIN) (2020, methodology): “More advanced variations can be found in [Das et al., 2018b; Yu et al., 2019; Wijmans et al., 2019].”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “It may give us some inspirations about how to build an embodied world model.”
ScanQA: 3D Question Answering for Spatial Scene Understanding (2021, background): “For example, the EQA dataset [17] contains 4,246 questions, consisting of 147 unique questions in its training set.”
Tool Learning with Foundation Models (2023, unknown): “Tasks typically assigned to agents include exploration (Ramakrishnan et al., 2021), navigation (Ye et al., 2021), question answering (Yu et al., 2019) within the simulated environment, or more interactive embodied task (Abramson et al., 2022) based on human instructions.”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “MT-EQA [225] focuses on complex questions involving multiple targets.”
A survey on deep multimodal learning for computer vision: advances, trends, applications, and datasets (2021, background): “13 Difference in results between EQA and VQA tasks: a EQA [90], b VQA [129]”