RGBD Datasets: Past, Present and Future

By Michael Firman. 2016

159 citations recorded by the source corpus.

Since the launch of the Microsoft Kinect, scores of RGBD datasets have been released. These have propelled advances in areas from reconstruction to gesture recognition. In this paper we explore the field, reviewing datasets across eight categories: semantics, object pose estimation, camera tracking, scene reconstruction, object tracking, human actions, faces and identification. By extracting relevant information in each category we help researchers to find appropriate data for their needs, and we consider which datasets have succeeded in driving computer vision forward and why. Finally, we examine the future of RGBD datasets. We identify key areas which are currently underexplored, and suggest that future directions may include synthetic data and dense reconstructions of static and dynamic scenes.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, background): “1A comprehensive and detailed overview of publicly-accessible RGBD datasets is given by [20] at http://www0.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, methodology): “We refer readers to the review article from Firman [14] for a comprehensive list.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, background): “Collecting and analyzing RGB-D imagery to train algorithms for scene understanding is an active area of research with great interest in computer vision, graphics, and robotics [10].”
SceneNN: A Scene Meshes Dataset with aNNotations (2016, methodology): “For a more comprehensive survey of publicly available datasets for other applications, we refer the readers to the excellent summary by Firman [9].”
Vision-based human activity recognition: a survey (2020, background): “Similarly, authors in [20, 46, 180, 181, 232] present a set of comprehensive reviews of the most commonly used RGB-D video-based activity recognition datasets.”
Deep Progressive Reinforcement Learning for Skeleton-Based Action Recognition (2018, background): “Kinect) and pose estimation algorithms [9], the amount of skeleton-based data is growing rapidly [10, 11].”
Multiview RGB-D Dataset for Object Instance Detection (2016, background): “More comprehensive discussion about additional RGB-D datasets can be found in [10].”
Campus3D: A Photogrammetry Point Cloud Benchmark for Hierarchical Understanding of Outdoor Scene (2020, background): “Firstly, the datasets may face various limitations from their sources which are are either RGB-D images [1, 9, 10, 28] or light detection and ranging (LiDAR) based mobile laser scanning (MLS) [3, 25, 26, 31] and terrestrial laser scanning (TLS) [14].”
Shading Annotations in the Wild (2017, background): “that synthetic scenes still cannot fully represent the complexity of natural images. Depth datasets. Several datasets contain RGB-D (depth) data, including NYUv2 [30], SUN RGB-D [31], and many others [12]. These datasets can be used to train algorithms to predict depth or surface normals from a sing”
Structured Prediction of Unobserved Voxels from a Single Depth Image (2016, background): “Depth datasets While large datasets such as NYU-Depth V2 [50] and the recently introduced SUN RGB-D dataset [53] exist for single depth images, few real world datasets are available containing complete 3D reconstructions [14].”
T-LESS: An RGB-D Dataset for 6D Pose Estimation of Texture-Less Objects (2017, background): “Only a few public RGB-D datasets, from over one hundred reported by Firman in [15], enable the evaluation of 6D object pose estimation methods.”
A dataset for developing and benchmarking active vision (2017, background): “[14] gives a list of a various RGB-D datasets, some focus on single objects [15], [16], in what we call “table-top” style data.”
Synthetic Data for Deep Learning (2019, background): “, the survey [176] and [116]), they cannot compete with synthetic data in terms of the quality of labeling and diversity of environments (see also Section 5.”