Deep Stereo: Learning to Predict New Views from the World's Imagery

By John Flynn, Ivan Neulander, James Philbin, Noah Snavely. 2015

702 citations recorded by the source corpus.

Deep networks have recently enjoyed enormous success when applied to recognition and classification problems in computer vision [22, 33], but their use in graphics problems has been limited ([23, 7] are notable recent exceptions). In this work, we present a novel deep architecture that performs new view synthesis directly from pixels, trained from a large number of posed image sets. In contrast to traditional approaches, which consist of multiple complex stages of processing, each of which requires careful tuning and can fail in unexpected ways, our system is trained end-to-end. The pixels from neighboring views of a scene are presented to the network, which then directly produces the pixels of the unseen view. The benefits of our approach include generality (we only require posed image sets and can easily apply our method to different domains), and high quality results on traditionally difficult scenes. We believe this is due to the end-to-end nature of our system, which is able to plausibly generate pixels according to color, depth, and texture priors learnt automatically from the training data. We show view interpolation results on imagery from the KITTI dataset [12], from data from [1] as well as on Google Street View images. To our knowledge, our work is the first to apply deep learning to the problem of new view synthesis from sets of real-world, natural imagery.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep multi-scale video prediction beyond mean square error (2015, background): “ted it on the NORB dataset. Beside unsupervised learning, a video predictive system may ﬁnd applications in robotics (Kosaka & Kak, 1992), video compression (Ascenso et al., 2005) and inpainting (Flynn et al., 2015) to name a few. Recently, predicting future video sequences appeared in different”
Photographic Image Synthesis with Cascaded Refinement Networks (2017, background): “Researchers have also studied image inpainting [32], superresolution [3, 17, 19], novel view synthesis [10, 45, 51], and interactive image manipulation [52].”
Dynamic Filter Networks (2016, methodology): “c, ﬁltering), as long as the transformation is implementable as a series of ﬁltering operations. This is illustrated in the ﬁrst experiment in the next section. Deep Stereo The deep stereo network of [5] can be seen as a speciﬁc instantiation of a dynamic ﬁlter network with a local ﬁltering layer wh”
OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “While 2D convolutional networks have proven very successful in extracting information from images [11, 13, 18, 35, 41, 42, 46, 48, 49], there exists comparably little work on processing three-dimensional data.”
Learning a Multi-View Stereo Machine (2017, background): “oposed in form of direct depth map regression [46, 31, 10], generating multiple depth maps from novel viewpoints [51], producing voxel occupancies [5, 16], geometry images [49] and point clouds [11]. [12] study a related problem of view interpolation, where a rough depth estimate is obtained within ”
View Synthesis by Appearance Flow (2016, methodology): “view extrapolation); 2) Unlike [27], who have a fixed number of input views, our multiview network is more flexible at both training and test time as it could take in an arbitrary number of input views for joint prediction, which is particularly beneficial when the number of input views varies at te”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, methodology): “In common with recent work [10, 14, 16], we use an end-to-end learning approach with view synthesis as the supervisory signal.”
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue (2016, background): “rning and depth estimation approach like ours could be extended at the test time to use a pair of images. There have been few works recently that approach the problem of novel view synthesis with CNN [32,10]. Deep stereo [10] uses a large set of posed images to learn a CNN that can interpolate betwe”
Video Frame Interpolation via Adaptive Separable Convolution (2017, methodology): “Flynn et al . developed a method that generates a novel image by projecting input images onto multiple depth planes and combining these depth planes to create the novel view [15].”
Neural RGB-D Surface Reconstruction (2021, methodology): “To reduce artifacts from classical reconstruction methods, a series of methods was proposed that use learned spatial priors to predict depth maps from color images [24, 28, 39], to learn multiview stereo using 3D CNNs on voxel grids [34, 68, 82], or multi-plane images [22], to reduce the influence o”
Unsupervised Monocular Depth Estimation with Left-Right Consistency (2016, background): “[13] introduced a novel image synthesis network called DeepStereo that generates new views by selecting pixels from nearby images.”
Neural Sparse Voxel Fields (2020, background): “Following-up works learned more 3D-structure aware representations and accompanying renderers (Flynn et al., 2016; Zhou et al., 2018; Mildenhall et al., 2019) with Multiplane Images (MPIs) as proxies, which only render a restricted range of novel views interpolating input views.”
3D Photography Using Context-Aware Layered Depth Inpainting (2020, methodology): “Classic image-based reconstruction and rendering techniques, however, require elaborate capture setups involving many images with large baselines [17, 59, 26, 45, 19, 12], and/or special hardware (e.”
FaceScape: A Large-Scale High Quality 3D Face Dataset and Detailed Riggable 3D Face Prediction (2020, background): “Predicting 3D shape from a single image is a key problem to many applications like view synthesis[22, 60, 61] and stereoscopic video generation[13, 25].”
D-NeRF: Neural Radiance Fields for Dynamic Scenes (2020, background): “thesizing novel photo-realistic views of dynamic scenes. For instance, [2,54,15] capture the dynamic scene at the same time instant from multiple views, to then generate 4D space-time visualizations. [11,32,53] also leverage on simultaneously capturing the scene from multiple cameras to estimate dep”
SuperDepth: Self-Supervised, Super-Resolved Monocular Depth Estimation (2018, background): “th [6], [5], [25], [16], and camera pose [7], [25]. Selfsupervised approaches thus typically focus on engineering the learning objective, for instance by treating view-synthesis as a proxy task [26], [27], [6], [8], [25], [28]. Related works also typically explore different architectures, for instan”