Dense Object Reconstruction with Semantic Priors

By Sid Ying-Ze Bao, Manmohan Chandraker, Yuanqing Lin, Silvio Savarese. 2013

115 citations recorded by the source corpus.

We present a dense reconstruction approach that overcomes the drawbacks of traditional multiview stereo by incorporating semantic information in the form of learned category-level shape priors and object detection. Given training data comprised of 3D scans and images of objects from various viewpoints, we learn a prior comprised of a mean shape and a set of weighted anchor points. The former captures the commonality of shapes across the category, while the latter encodes similarities between instances in the form of appearance and spatial consistency. We propose robust algorithms to match anchor points across instances that enable learning a mean shape for the category, even with large shape variations across instances. We model the shape of an object instance as a warped version of the category mean, along with instance-specific details. Given multiple images of an unseen instance, we collate information from 2D object detectors to align the structure from motion point cloud with the mean shape, which is subsequently warped and refined to approach the actual shape. Extensive experiments demonstrate that our model is general enough to learn semantic priors for different object categories, yet powerful enough to reconstruct individual shapes with large variations. Qualitative and quantitative evaluations show that our framework can produce more accurate reconstructions than alternative state-of-the-art multiview stereo systems.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Learning a Multi-View Stereo Machine (2017, methodology): “Learning has been used for multi-view reconstruction in the form of shape priors for objects [2, 9, 58, 20, 27, 52], or semantic class speciﬁc surface priors for scenes [22, 17, 45].”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “For such cases object shape priors have been proposed [8], [14], [36].”
Lifting Object Detection Datasets into 3D (2015, positive): “To account for differences between the 3D exemplars and the object depicted, Su and Guibas [26] choose a few exemplars that appear to be similar to the depicted shape and combine them to produce a single depth map.”
Patches, Planes and Probabilities: A Non-Local Prior for Volumetric 3D Reconstruction (2016, background): “, by constraining the set of plausible geometries [2, 11, 19] or by modeling class specific normal distributions [20,21].”
WarpNet: Weakly Supervised Matching for Single-View Reconstruction (2016, methodology): “Recently, methods have been proposed to match instances across categories, relying on supervision in the form of part (keypoint) annotations [7, 15, 29] or 3D CAD models [3, 8] to augment appearance information with shape priors.”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, methodology): “In addition to mean shape, Bao et al. (2013) propose to learn a set of anchor points as representative”
Deep Marching Cubes: Learning Explicit Surface Representations (2018, background): “Unfortunately, the expressiveness of such prior models is limited to simple local smoothness assumptions [2, 17, 21, 47] or very specialized shape models [1, 15, 16, 42].”
Joint 3d Estimation of Vehicles and Scene Flow (2015, methodology): “(Bao et al., 2013) compute a mean shape of the observed object class along with a set of discrete anchor points.”
Dense Semantic 3D Reconstruction (2017, methodology): “Concurrently to the original research [13] that lead to this paper, a method using semantic information for dense object reconstruction in form of shape priors [1] has been developed.”
Mesh R-CNN (2019, background): “There is a broad line of work on multi-view reconstruction of objects and scenes, from classical binocular stereo [17, 53] to using shape priors [1, 2, 6, 21] and modern learning techniques [24, 26, 54].”
Class Specific 3D Object Shape Priors Using Surface Normals (2014, background): “Object shape priors have recently attained much attention in the field of dense 3D reconstruction [3, 5].”
Displets: Resolving stereo ambiguities using object knowledge (2015, background): “In contrast to [1,7], the number of objects is unknown in our case.”
Joint 3D Scene Reconstruction and Class Segmentation (2013, methodology): “The label assignments will be encoded with L + 1 indicator functions x i : Ω → [0] [1], i ∈ {0, . . . , L}: x i (z) = 1 iff label i is assigned at z ∈ Ω. Note that in the following, the dependence of all quantities on the 3D location z will be indicated with a subscript to be more consistent with…”
Semantic Multi-view Stereo: Jointly Estimating Objects and Voxels (2017, methodology): “densify multi-view stereo point clouds by fitting 3D shape models [3].”
Learning 3D Shape Completion from Laser Scan Data with Weak Supervision (2018, background): “tion as optimization problem over the corresponding (lowerdimensional) latent space [3, 10, 13, 22].”