An Exploration of Embodied Visual Exploration

By Santhosh K. Ramakrishnan, Dinesh Jayaraman, Kristen Grauman. 2020

113 citations recorded by the source corpus.

Embodied computer vision considers perception for robots in novel, unstructured environments. Of particular importance is the embodied visual exploration problem: how might a robot equipped with a camera scope out a new environment? Despite the progress thus far, many basic questions pertinent to this problem remain unanswered: (i) What does it mean for an agent to explore its environment well? (ii) Which methods work well, and under which assumptions and environmental settings? (iii) Where do current approaches fall short, and where might future work seek to improve? Seeking answers to these questions, we first present a taxonomy for existing visual exploration algorithms and create a standard framework for benchmarking them. We then perform a thorough empirical study of the four state-of-the-art paradigms using the proposed framework with two photorealistic simulated 3D environments, a state-of-the-art exploration architecture, and diverse evaluation metrics. Our experimental results offer insights and suggest new performance metrics and baselines for future work in visual exploration. Code, models and data are publicly available.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

A Survey of Embodied AI: From Simulators to Research Tasks (2021, methodology): “Visual exploration has a few common baselines [48].”
Tool Learning with Foundation Models (2023, unknown): “Tasks typically assigned to agents include exploration (Ramakrishnan et al., 2021), navigation (Ye et al., 2021), question answering (Yu et al., 2019) within the simulated environment, or more interactive embodied task (Abramson et al., 2022) based on human instructions.”
A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Tasks (2020, background): “Methods also investigate the utility of topological and latent memory maps [35,78,37,99], graph-4 based learning [99,103], meta-learning [98], unimodal baselines [90], 3D point clouds [97], and eﬀective exploration [95,78,16,74] to improve embodied navigational agents.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, methodology): “ScenePriors [65] NavExp [49] IntExp [39] Ours Fig.”
Learning Affordance Landscapes for Interaction Exploration in 3D Environments (2020, background): “• OBJCOVERAGE [18, 47] rewards an agent for visiting new objects (moving close to it, and centering it in view), but not for interacting with them.”
Diffusion-based Generation, Optimization, and Planning in 3D Scenes (2023, methodology): “However, existing work relies heavily on model-based planners with the single-step dynamic model [5,11,48,67,74,80], lacking a trajectory-level optimization for long-horizon planning.”
Occupancy Anticipation for Efficient Exploration and Navigation (2020, methodology): “Prior work on exploration [11,17,10,52] often uses area-coverage—the area seen in the environment during navigation—as a reward function to guide exploration.”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Exploration methods are usually guided by intrinsic rewards that are selfgenerated by the robotic agent, and are not specific to any task [9, 47, 49, 60, 69, 81, 85, 88, 93, 110].”