Embodied Question Answering in Photorealistic Environments With Point Cloud Perception

By Erik Wijmans, Samyak Datta, Oleksandr Maksymets, Abhishek Das, Georgia Gkioxari, Stefan Lee, Irfan Essa, Devi Parikh, Dhruv Batra. 2019

205 citations recorded by the source corpus.

To help bridge the gap between internet vision-style problems and the goal of vision for embodied perception we instantiate a large-scale navigation task -- Embodied Question Answering [1] in photo-realistic environments (Matterport 3D). We thoroughly study navigation policies that utilize 3D point clouds, RGB images, or their combination. Our analysis of these models reveals several key findings. We find that two seemingly naive navigation baselines, forward-only and random, are strong navigators and challenging to outperform, due to the specific choice of the evaluation setting presented by [1]. We find a novel loss-weighting scheme we call Inflection Weighting to be important when training recurrent models for navigation with behavior cloning and are able to out perform the baselines with this technique. We find that point clouds provide a richer signal than RGB images for learning obstacle avoidance, motivating the use (and continued study) of 3D deep learning models for embodied navigation.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Question Answering & Reading Comprehension, 3D Vision & Pose, Robotics & Control

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Question Answering & Reading Comprehension

In 3D Vision & Pose

In Robotics & Control

Citations in the corpus

Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learning (2019, background): “Nevertheless, modeling human assistance in these problems remains simplistic (Table 1): they either do not incorporate the ability to request additional help while executing tasks (Misra et al., 2014, 2017; Anderson et al., 2018b; Chen et al., 2019; Das et al., 2018; Misra et al., 2018; Wijmans et a”
COPHY: Counterfactual Learning of Physical Dynamics (2019, background): “, 2017), embodied AI (Wijmans et al., 2019), as well as learning intuitive physics (Lerer et al.”
Visually-Grounded Planning without Vision: Language Models Infer Detailed Plans from High-level Instructions (2020, background): “…environments with steadily increasing ﬁdelity are allowing virtual agents to learn to perform high-level tasks that couple language understanding, visual planning, and embodied reasoning through sensorimotor grounded representations (Gordon et al., 2018; Puig et al., 2018; Wijmans et al., 2019).”
Beyond the Nav-Graph: Vision-and-Language Navigation in Continuous Environments (2020, background): “As described in [30], inflection weighting places emphasis on time-steps where actions change (i.”
OpenEQA: Embodied Question Answering in the Era of Foundation Models (2024, unknown): “They focus on leveraging RGB-D to accomplish navigation tasks in simulation [41], in which the agent must seek out multiple target locations or objects sampled from a closed vocabulary set [50] Our work takes inspiration from such prior works [7] and modernizes it to be relevant in the current era…”
Aligning Cyber Space With Physical World: A Comprehensive Survey on Embodied AI (2024, unknown): “MP3D-EQA [138] is built on a simulator developed based on MINOS [148] using the Authorized licensed use limited to the terms of the applicable license agreement with IEEE.”
Audio-Visual Embodied Navigation (2019, background): “Visual navigation can be tied with other tasks to attain intelligent behavior, such as question answering [23, 25, 39, 99], active visual recognition [50, 110, 109], instruction following [88, 6, 86, 17, 16] and collaborative tasks with visual inputs [49, 48, 24].”
Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration (2020, background): “single agent settings for tasks such as navigation (Savva et al., 2019; Xia et al., 2018; Brodeur et al., 2017; Zhu et al., 2017; Xia et al., 2018) , embodied question answering (Gordon et al., 2017; Wijmans et al., 2019; Das et al., 2018), or single agent task completion (Puig et al., 2018; Shridha”
Point-Bind & Point-LLM: Aligning Point Cloud with Multi-modality for 3D Understanding, Generation, and Instruction Following (2023, background): “Another series of works connects descriptive natural language with 3D data, which is applied to 3D captioning [12, 87], question answering [3, 78], and visual grounding [24, 79].”
A Survey on Vision--Language--Action Models for Embodied AI (2024, unknown): “MP3D-EQA [226] converts previous RGB inputs to point clouds, testing 3D perception capabilities.”
EgoVQA - An Egocentric Video Question Answering Benchmark Dataset (2019, background): “Perhaps the most related work are EmbodiedQA [18,53] which proposed to use robot agent to navigate the environment and fulfill the query intents.”
Auxiliary Tasks Speed Up Learning PointGoal Navigation (2020, unknown): “Su-pervised auxiliary tasks expose privileged information to the agent (such as depth [22, 27, 28, 29]).”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Embodied navigation also aids AI agents to develop behavior such as instruction following [38,4,82,95,3], city navigation [18,64,63,94], question answering [21,22,34,97,24], and active visual recognition [105,104].”
Simultaneous Mapping and Target Driven Navigation (2019, unknown): “[15, 19, 27, 29, 26, 6, 13, 24, 25].”
An Exploration of Embodied Visual Exploration (2020, unknown): “We use inflection weighting to prevent the policy from simply repeating the previous action [70].”
DD-PPO: Learning Near-Perfect PointGoal Navigators from 2.5 Billion Frames (2019, background): “Specifically, over the last few years, a large number of works have proposed training virtual robots (or ‘embodied agents’) in rich 3D simulators before transferring the learned skills from simulation to reality (Beattie et al., 2016; Chaplot et al., 2017; Das et al., 2018; Gordon et al., 2018; Ande”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, background): “gation [1], to grounded instruction following [2, 23] and question answering [8, 11, 27].”
ScanQA: 3D Question Answering for Spatial Scene Understanding (2021, unknown): “Embodied question answering (EQA) [17, 46, 49] is a combination of visual question answering and navigation such as vision-and-language navigation tasks [5, 11, 43, 47] and models [18, 28].”