SemanticPaint

By Julien P. C. Valentin, Vibhav Vineet, Ming-Ming Cheng, David Kim, Jamie Shotton, Pushmeet Kohli, Matthias Nießner, Antonio Criminisi, Shahram Izadi, Philip H. S. Torr. 2015

104 citations recorded by the source corpus.

We present a new interactive and online approach to 3D scene understanding. Our system, SemanticPaint, allows users to simultaneously scan their environment whilst interactively segmenting the scene simply by reaching out and touching any desired object or surface. Our system continuously learns from these segmentations, and labels new unseen parts of the environment. Unlike offline systems where capture, labeling, and batch learning often take hours or even days to perform, our approach is fully online. This provides users with continuous live feedback of the recognition during capture, allowing to immediately correct errors in the segmentation and/or learning—a feature that has so far been unavailable to batch and offline methods. This leads to models that are tailored or personalized specifically to the user's environments and object classes of interest, opening up the potential for new applications in augmented reality, interior design, and human/robot navigation. It also provides the ability to capture substantial labeled 3D datasets for training large-scale visual recognition systems.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Revisiting Point Cloud Classification: A New Benchmark Dataset and Classification Model on Real-World Data (2019, methodology): “ShapeNet [7] is an alternative large-scale dataset of 3D CAD shapes with approximately 51 , 000 objects in 55 categories.”
ScanNet: Richly-Annotated 3D Reconstructions of Indoor Scenes (2017, methodology): “This could be achieved with a volumetric CRF [64], as used in [86]; however, our goal in this task to focus exclusively on the per-voxel classification accuracy.”
Matterport3D: Learning from RGB-D Data in Indoor Environments (2017, methodology): “These previous RGB-D datasets have been used to train models for several standard scene understanding tasks, including semantic segmentation [27, 33, 13, 40, 7], 3D object detection [31, 24, 14, 36, 37], normal estimation [41, 23, 9, 3], camera relocalization [30, 39], and others [47, 11, 12].”
3DMV: Joint 3D-Multi-View Prediction for 3D Semantic Scene Segmentation (2018, background): “The analog task can be formulated in 3D, where the goal is to predict semantic labels on a per-voxel level [31,32].”
Pointwise Convolutional Neural Networks (2017, methodology): “In the 3D domain, interactive semantic segmentation [40, 39] relied on user strokes to propagate segmentation.”