Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency

By Shubham Tulsiani, Tinghui Zhou, Alyosha A. Efros, Jitendra Malik. 2017

590 citations recorded by the source corpus.

We study the notion of consistency between a 3D shape and a 2D observation and propose a differentiable formulation which allows computing gradients of the 3D shape given an observation from an arbitrary view. We do so by reformulating view consistency using a differentiable ray consistency (DRC) term. We show that this formulation can be incorporated in a learning framework to leverage different types of multi-view observations e.g. foreground masks, depth, color images, semantics etc. as supervision for learning single-view 3D prediction. We present empirical analysis of our technique in a controlled setting. We also show that this approach allows us to improve over existing techniques for single-view reconstruction of objects from the PASCAL VOC dataset.

Subfields: Computational Mechanics, Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition, 3D Vision & Pose

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

In 3D Vision & Pose

Citations in the corpus

Occupancy Networks: Learning 3D Reconstruction in Function Space (2018, background): “Early works have considered the problem of reconstructing 3D geometry from a single image using 3D convolutional neural networks which operate on voxel grids [9, 68, 77].”
Learning to Reconstruct Shapes from Unseen Classes (2018, methodology): “The performance drop from object-centered DRC to viewer-centered DRC supports this empirically.”
Learning Shape Priors for Single-View 3D Completion and Reconstruction (2018, methodology): “researchers have made impressive progress on the these tasks [7,52,8], making use of gigantic 3D datasets [5,60,59].”
RayNet: Learning Volumetric 3D Reconstruction with Ray Potentials (2018, background): “[32] relax these assumptions by formulating a differentiable view consistency loss that measures the inconsistency between the predicted 3D shape and its observation.”
Pix3D: Dataset and Methods for Single-Image 3D Shape Modeling (2018, methodology): “We compare our model, with and without the pose estimation branch, with the state-ofthe-art systems, including 3D-VAE-GAN [62], 3D-R2N2 [9], Ours (w/ Pose) Ours (w/o Pose) DRC [59], and MarrNet [60].”
Learning Category-Specific Mesh Reconstruction from Image Collections (2018, methodology): “We compare to previous deformable model fittingbased [15] and volumetric prediction [30] approaches that use similar image collection supervision.”
MarrNet: 3D Shape Reconstruction via 2.5D Sketches (2017, methodology): “These approaches usually regress a 3D shape from a single RGB image directly [Tulsiani et al., 2017, Choy et al., 2016, Wu et al., 2016b].”
Learning a Multi-View Stereo Machine (2017, background): “A line of recent works, complementary to ours, has proposed to incorporate ideas from multi-view geometry in a learning framework to train single view prediction systems [14, 56, 53, 42, 61] using multiple views as supervisory signal.”
Hierarchical Surface Prediction for 3D Object Reconstruction (2017, background): “ground truth shapes, CNNs can also be trained to predict 3D shapes using weaker supervision [27], [35], [34] for example image silhouettes.”
3D Shape Induction from 2D Views of Multiple Objects (2016, background): “[31] learn a model to map images to 3D shape provided with color images or silhouettes of objects taken from”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, background): “, [13, 41, 46, 52]) has also shown success in learning 3D volumetric representations from 2D observations based on similar principles of projective geometry.”
Neural 3D Mesh Renderer (2017, methodology): “are 3D extensions of pixels, are the most widely used format in machine learning because they can be processed by CNNs [2, 17, 20, 24, 30, 31, 34, 35, 36].”
Unsupervised Learning of 3D Object Categories from Videos in the Wild (2021, background): “NeRF [36] has raised the interest in neural scene representation due to its high-quality output, inspired by positional encoding proposed in [57] and differentiable volume rendering from [19, 55].”
3D Object Dense Reconstruction from a Single Depth View (2018, background): “For general and complex object reconstruction from a single RGB image, recent works [52] [53] [54] aim to infer 3D shapes using multiple RGB images for weak supervision.”
Stereo magnification (2018, background): “Finally, a variety of work in computer vision has used view synthesis as an indirect form of supervision for other tasks, such as predicting depth, shape, or optical flow from one or more images [Garg and Reid 2016; Godard et al. 2017; Liu et al. 2017; Tulsiani et al. 2017; Vijayanarasimhan et al. 2”
Soft Rasterizer: Differentiable Rendering for Unsupervised Single-View Mesh Reconstruction (2019, methodology): “When considering reconstructing 3D shape, voxel-based representation has received most attention [52, 56, 44, 48] due to its simplicity of regular structure and compatibility with convolutional neural network.”