A dataset for developing and benchmarking active vision

By Phil Ammirato, Patrick Poirson, Eunbyung Park, Jana Kosecka, Alexander C. Berg. 2017

209 citations recorded by the source corpus.

We present a new public dataset with a focus on simulating robotic vision tasks in everyday indoor environments using real imagery. The dataset includes 20,000+ RGB-D images and 50,000+ 2D bounding boxes of object instances densely captured in 9 unique scenes. We train a fast object category detector for instance detection on our data. Using the dataset we show that, although increasingly accurate and fast, the state of the art for object detection is still severely impacted by object scale, occlusion, and viewing direction all of which matter for robotics applications. We next validate the dataset for simulating active vision, and use the dataset to develop and evaluate a deep-network-based system for next best move prediction for object classification using reinforcement learning. Our dataset is available for download at cs.unc.edu/∼ammirato/active_vision_dataset_website/.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Evaluation & Benchmarks

Citations in the corpus

Vision-and-Language Navigation: Interpreting Visually-Grounded Navigation Instructions in Real Environments (2017, background): “In seeking to strike a more favorable balance between interactivity and visual realism, we share similar motivations with the much smaller Active Vision Dataset [2], which is also based on densely sampled real-world images.”
Cognitive Mapping and Planning for Visual Navigation (2017, background): “Numerous works such as (Chang et al. 2017; Dai et al. 2017; Ammirato et al. 2017) have collected large-scale datasets consisting of scans of real world environments, while (Savva et al. 2017; Wu et al. 2018; Xia et al. 2018) have built more sophisticated simulation environments based on such scans.”
Habitat: A Platform for Embodied AI Research (2019, background): “Datasets have been a key driver of progress in computer vision, NLP, and other areas of AI [10, 18, 4, 1].”
Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detection (2017, methodology): “Depending on the type of data, human annotations can be augmented with object tracking or 3D sensor information [1, 13, 24, 46, 52].”
Gibson Env: Real-World Perception for Embodied Agents (2018, background): “are of limited use when it comes to active agent. Similarly, video datasets [57,70,101] are pre-recorded and thus passive. Virtual environments have been a remedy for this, classically [68] and today [106,36,31,83,47,41,11,9,72, 8,98]. Computer games, e.g. Minecraft [49], Doom [51] and GTA5 [69] hav”
Emergence of exploratory look-around behaviors through active observation completion (2019, background): “object recognition system (8, 10, 11, 26, 30) can only see its previously selected views of the object; yet, if trained with CAD models, it could observe all possible views while learning.”
An Exploration of Embodied Visual Exploration (2020, unknown): “rd motion 0.3 m 0.25 m Rotation angle 30 10 Table 1. The contrasting properties of AVD and MP3D provide diverse testing conditions. Last 2 rows show the action magnitudes. The Active Vision Dataset [2] is a dataset of dense RGB-D scans from 15 unique indoor houses and ofﬁce buildings. We simulate em”
InteriorNet: Mega-scale Multi-sensor Photo-realistic Indoor Scenes Dataset (2018, background): “In the active vision dataset [6], limited control options are provided to control the trajectory in 2D while navigating through the images.”
Semantic Curiosity for Active Visual Learning (2020, unknown): “This has been instantiated in the context of object detection [1], amodal object detection [56], scene completion [29], and localization [11,22].”
Visual Representations for Semantic Target Driven Navigation (2018, methodology): “All the evaluations are done on AVD because we want to see the effect of representation on the real data.”
Learning to Look Around: Intelligently Exploring Unseen Environments for Unknown Tasks (2017, background): “Arbitrarily framed snapshots of the visual world are ill-suited both for human perception [16, 49] and for machine perception [3, 70].”
SoundSpaces: Audio-Visual Navigation in 3D Environments (2019, background): “Their realistic meshes can be rendered from agent-selected viewpoints to train and test RL policies for navigation in a reproducible manner [3,13,104,53,7,83,10,103,59].”
Audio-Visual Embodied Navigation (2019, background): “Their realistic meshes can be rendered from agent-selected viewpoints to train and test RL policies for navigation in a reproducible manner [4, 13, 108, 56, 8, 85, 11, 107, 61].”
Sidekick Policy Learning for Active Visual Exploration (2018, background): “Similarly, an active object recognition system [29,31,2,65,28] can only see its previously selected views of the object; yet if trained with CAD models, it could observe all possible views while learning.”
MapNet: An Allocentric Spatial Memory for Mapping Environments (2018, methodology): “This is explained by the fact that classic SLAM systems rely heavily on temporal continuity, and AVD has a very low frame-rate.”