Building a database of 3D scenes from user annotations

By Bryan C. Russell, Antonio Torralba. 2009

101 citations recorded by the source corpus.

In this paper, we wish to build a high quality database of images depicting scenes, along with their real-world three-dimensional (3D) coordinates. Such a database is useful for a variety of applications, including training systems for object detection and validation of 3D output. We build such a database from images that have been annotated with only the identity of objects and their spatial extent in images. Important for this task is the recovery of geometric information that is implicit in the object labels, such as qualitative relationships between objects (attachment, support, occlusion) and quantitative ones (inferring camera parameters). We describe a model that integrates cues extracted from the object labels to infer the implicit geometric information. We show that we are able to obtain high quality 3D information by evaluating the proposed approach on a database obtained with a laser range scanner. Finally, given the database of 3D scenes, we show how it can find better scene matches for an unlabeled image by expanding the database through viewpoint interpolation to unseen views.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

ShapeNet: An Information-Rich 3D Model Repository (2015, methodology): “Though we ﬁrst use WordNet due to its popularity, the ShapeNet UI is designed to allow multiple views into the collection of shapes that it contains, including different taxonomy views and faceted navigation.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, background): “Notable attempts include LabelMe3D [46], NYU-D [50], SUN RGB-D [53], KITTI [16], and modern large-scale RGB-D scene datasets [10, 40, 54].”
Lifting Object Detection Datasets into 3D (2015, methodology): “Morphable models are low-dimensional parametric models that have been used to represent the shape of many object classes.”
Deep convolutional neural fields for depth estimation from a single image (2014, neutral): “, user annotations [6], semantic labellings [7,8].”
Learning Depth from Single Monocular Images Using Deep Convolutional Neural Fields (2015, background): “In recent years, efforts have been made towards incorporating additional sources of information, e.g., user annotations [8], semantic labellings [9], [1].”
Learning invariance through imitation (2011, background): “The first is a probabilistic approach that is suitable for batch learning.”
Towards unified depth and semantic prediction from a single image (2015, background): “In the depth estimation literature, semantic information has long been used as a high-level guidance [14] [15] [23] [11] [29].”
LabelMe: Online Image Annotation and Applications (2010, unknown): “Please see [28], [48] for more details.”
Three-Dimensional Object Detection and Layout Prediction Using Clouds of Oriented Gradients (2016, background): “A growing number of annotated RGB-D datasets have been constructed to train and evaluate indoor scene understanding methods [30, 21, 34, 35].”
Context models and out-of-context objects (2012, methodology): “Our model consists of two tree-structured graphical models: the first part relates object categories using cooccurrence statistics, and the second part relates detector outputs using support relationships.”
Reconstructing PASCAL VOC (2014, background): “In a related approach, [22] targeted the problem of automatically bootstrapping 3D scene geometry from 2D annotations on the LabelMe dataset — instead, we focus on objects.”
ObjectNet3D: A Large Scale Database for 3D Object Recognition (2016, background): “An influential work in building datasets with 2D-3D alignment is LabelMe3D [27] that estimates the 3D scene structure based on user annotations.”
Estimating Depth From Monocular Images as Classification Using Deep Fully Convolutional Residual Networks (2016, background): “To name a few, Russell and Torralba [19] estimated depths through user annotations.”
Scene Parsing with Object Instances and Occlusion Ordering (2014, background): “Fortunately, our datasets come from LabelMe where the ground truth consists of overlapping object polygons, and it allows us to learn such relationships [17].”
Indoor Segmentation and Support Inference from RGBD Images (2012, background): “Russell and Torralba [9] show how a dataset of user-annotated scenes can be used to infer 3D structure and support; our approach, in contrast, is fully automatic.”
3D Scene Graph: A Structure for Unified Semantics, 3D Space, and Camera (2019, methodology): “[41] employs object segmentation masks and labels from 2D annotations to automatically recover the 3D scene geometry.”
Amodal Completion and Size Constancy in Natural Scenes (2015, background): “[23] also tackled a problem similar to operationalizing size constancy and inferred object sizes of annotated objects.”