SUN3D: A Database of Big Spaces Reconstructed Using SfM and Object Labels

By Jianxiong Xiao, Andrew Owens, Antonio Torralba. 2013

849 citations recorded by the source corpus.

Existing scene understanding datasets contain only a limited set of views of a place, and they lack representations of complete 3D spaces. In this paper, we introduce SUN3D, a large-scale RGB-D video database with camera pose and object labels, capturing the full 3D extent of many places. The tasks that go into constructing such a dataset are difficult in isolation -- hand-labeling videos is painstaking, and structure from motion (SfM) is unreliable for large spaces. But if we combine them together, we make the dataset construction task much easier. First, we introduce an intuitive labeling tool that uses a partial reconstruction to propagate labels from one frame to another. Then we use the object labels to fix errors in the reconstruction. For this, we introduce a generalization of bundle adjustment that incorporates object-to-object correspondences. This algorithm works by constraining points for the same object from different frames to lie inside a fixed-size bounding box, parameterized by its rotation and translation. The SUN3D database, the source code for the generalized bundle adjustment, and the web-based 3D annotation tool are all available at http://sun3d.cs.princeton.edu.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Hypersim: A Photorealistic Synthetic Dataset for Holistic Indoor Scene Understanding (2020, neutral): “This distribution is roughly consistent with existing indoor depth datasets [64, 70, 78], but is noticeably different from outdoor datasets [16, 23].”
Generative Image Modeling Using Style and Structure Adversarial Networks (2016, methodology): “We perform the experiments on scene classification on SUN RGB-D dataset [61,62,63,14] as well as object detection on NYUv2 dataset.”
RGB-D datasets using microsoft kinect or similar sensors: a survey (2016, methodology): “The curves under the images are the accelerometer data at 50 Hz of devices attached to the knife, the mixing spoon, the small spoon, the peeler, the glass, the oil bottle, and the pepper dispenser mentioned datasets above, another 26 extra RGB-D datasets for different applications are also added int”
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, background): “The semantic seg-mentations are more than an order of magnitude larger than any previous RGB-D dataset.”
Volumetric and Multi-view CNNs for Object Classification on 3D Data (2016, background): “, [26, 31, 28] – this is the first containing hundreds of annotated models from dense 3D reconstructions.”
ShapeNet: An Information-Rich 3D Model Repository (2015, background): “ons beyond category labels. Other commonly-used datasets contain segmentations [2], correspondences [13,12], hierarchies [19], symmetries [11], salient features [3], semantic segmentations and labels [36], alignments of 3D models with images [35], semantic ontologies [5], and other functional annota”
A Large Dataset to Train Convolutional Networks for Disparity, Optical Flow, and Scene Flow Estimation (2015, methodology): “Dataset MPI Sintel [2] KITTI Benchmark Suite [16] SUN3D[26] NYU2[21] Ours 2012 2015 FlyingThings3D Monkaa Driving”
SUN Database: Exploring a Large Collection of Scene Categories (2014, background): “Future works include going beyond 2D images and reasoning about scenes in 3D (Xiao et al. 2013; Zhang et al. 2014; Song and Xiao 2014).”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “Previous RGB-D datasets have provided data for single rooms or small sets of adjacent rooms [44, 7], or single ﬂoors of a building [2].”
Orientation-boosted Voxel Nets for 3D Object Recognition (2016, positive): “rom range-scanners, such as the Sydney Urban Objects used in [17]. The Sydney dataset is of special interest in our work as it lets us compare our results to the work of [2]. SUN-RGBD [18] and SUN-3D [19] gather some Reconstructed point-clouds and RGB-D datasets in one place and in some cases they a”
Spatio-temporal video autoencoder with differentiable memory (2015, background): “Sun3D Xiao et al. (2013) contains video sequences with labels obtained by label propagation.”
Image Segmentation Using Deep Learning: A Survey (2020, background): “SUN-3D [167] is a large RGB-D video dataset that contains 415 sequences captured from 254 different spaces in 41”
A Field Model for Repairing 3D Shapes (2016, background): “Recent advances of 3D acquisition devices and 3D scene reconstruction research [28, 38, 39, 40, 4] have enabled large-scale acquisition of 3D scene data and this has raised”
RDFNet: RGB-D Multi-level Residual Feature Fusion for Indoor Semantic Segmentation (2017, background): “The dataset consists of 10335 pairs of RGB and depth images captured from four different depth sensors, which contains images from NYUDv2 depth [35], Berkeley B3DO [16], and SUN3D [40], as well as newly captured images.”
Synthetic Data for Deep Learning (2019, background): “There are large-scale efforts to create real annotated datasets of indoor scenes [92, 133, 548, 556, 652, 654], but synthetic data is increasingly being used in the field [693].”
3D Semantic Parsing of Large-Scale Indoor Spaces (2016, background): “Notice that these classes are more fine-grained and challenging than many of the semantic indoor segmentation datasets [31, 39].”
AdaBins: Depth Estimation Using Adaptive Bins (2020, background): “SUN RGB-D is an indoor dataset consisting of around 10K images with high scene diversity collected with four different sensors [38, 43, 22].”
SceneNN: A Scene Meshes Dataset with aNNotations (2016, methodology): “DOI 10.1109/3DV.2016.18 92 probably most similar to the SUN3D dataset created by Xiao et al. [36].”
Depth Estimation via Affinity Learned with Convolutional Spatial Propagation Network (2018, background): “Given a single image, recent efforts to estimate per-pixel depths have yielded highquality outputs by taking advantage of deep fully convolutional neural networks [3,4] and large amount of training data from indoor [1,5,6] and outdoor [2,7,8].”