Deep3D: Fully Automatic 2D-to-3D Video Conversion with Deep Convolutional Neural Networks

By Junyuan Xie, Ross B. Girshick, Ali Farhadi. 2016

447 citations recorded by the source corpus.

As 3D movie viewing becomes mainstream and the Virtual Reality (VR) market emerges, the demand for 3D contents is growing rapidly. Producing 3D videos, however, remains challenging. In this paper we propose to use deep neural networks to automatically convert 2D videos and images to a stereoscopic 3D format. In contrast to previous automatic 2D-to-3D conversion algorithms, which have separate stages and need ground truth depth map as supervision, our approach is trained end-to-end directly on stereo pairs extracted from existing 3D movies. This novel training scheme makes it possible to exploit orders of magnitude more data and significantly increases performance. Indeed, Deep3D outperforms baselines in both quantitative and human subject evaluations.

Subfields: Computer Graphics and Computer-Aided Design, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Graphics and Computer-Aided Design

In Computer Vision and Pattern Recognition

Citations in the corpus

OctNet: Learning Deep 3D Representations at High Resolutions (2016, background): “While 2D convolutional networks have proven very successful in extracting information from images [11, 13, 18, 35, 41, 42, 46, 48, 49], there exists comparably little work on processing three-dimensional data.”
Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks (2016, background): “More recently, works in image synthesis have begun to produce impressive results by training variants of neural network structures to produce novel images [Gregor et al., 2015, Xie et al., 2016].”
Unsupervised Learning of Depth and Ego-Motion from Video (2017, background): “Concurrently, Deep3D [51] predicts a second stereo viewpoint from an input image using stereoscopic film footage as training data.”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “We note that 3D movies have been used in related tasks in isolation [18, 46].”
Unsupervised CNN for Single View Depth Estimation: Geometry to the Rescue (2016, background): “There have been few works recently that approach the problem of novel view synthesis with CNN [32,10].”
Video Frame Interpolation via Adaptive Separable Convolution (2017, background): “Xie et al . developed a neural network that synthesizes an extra view from a monocular video to convert it to a stereo video [54].”
DynamoNet: Dynamic Action and Motion Network (2019, background): “Given an observed image or a sequence of frames predict the future frames is very popular these days [15, 18, 33, 40, 44, 60, 61, 66], where mostly researchers predict low-level pixels or motion, until recently image synthesis is done via neural networks: Generative adversarial networks [5, 17, 34, ”
Digging Into Self-Supervised Monocular Depth Estimation (2018, background): “[67] proposed such a model with discretized depth for the problem of novel view synthesis.”
Artificial Neural Networks-Based Machine Learning for Wireless Networks: A Tutorial (2017, methodology): “gorithm. [170]. collection. -to-ground ) algorithm. [171]. data. Vs ) algorithm. ork ) algorithm. VR [176]. data. ement ) algorithm. [178]. resources. formats ) algorithm. [179]. ANNs. decoding ) algorithm. [181]. images ) algorithm. [182]. prediction ) algorithm. correlation ) ”
Semi-Supervised Deep Learning for Monocular Depth Map Prediction (2017, background): “[27] do not regress the depth maps directly, but produce probability maps for different disparity levels.”
Unsupervised Learning of Depth and Ego-Motion from Monocular Video Using 3D Geometric Constraints (2018, background): “own transformation, the depth can be learned from that functional relationship (plus some regularization). Other novel learning approaches, that also need more than one image for depth estimation are [27, 20, 15, 14, 28]. Godard et al. [12] offer an approach to learn single-view depth estimation usi”
Video Frame Synthesis Using Deep Voxel Flow (2017, background): “Generative CNNs can also be used to generate new views of a scene from existing photos taken at nearby viewpoints [7, 35].”
Deep Ordinal Regression Network for Monocular Depth Estimation (2018, methodology): “Thanks to multi-level contextual and structural information from powerful very deep networks ( e.g ., VGG [56] and ResNet [24]), depth estimation has been boosted to a new accuracy level [11, 17, 33, 35, 59].”
Artificial intelligence in the creative industries: a review (2020, methodology): “DL-based methods also provide state-of-the-art performance for generating the corresponding right view from a left view in a stereo pair [292], and for converting 2D face images to 3D face reconstructions in [200–202] (Fig.”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “Self-supervised methods remove the limitation of requiring ground truth depth supervision, instead training with image-reconstruction losses using stereo pairs [87, 21, 23] or monocular video sequences [99].”
Machine Learning for Wireless Networks with Artificial Intelligence: A Tutorial on Neural Networks (2017, methodology): “In [179], a deep neural network based algorithm is proposed to automatically convert 2D videos and images to a stereoscopic 3D format.”
Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding (2018, background): “Some of them relies on stereo image pairs [9], [50], [51], e.”
From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation (2019, methodology): “Therefore, a number of techniques, for example, multi-scale networks [29, 11], skip connections [17, 45] or multi-layer deconvolutional networks [25, 15, 24] are applied to consolidate feature maps from higher resolutions.”
Deep Virtual Stereo Odometry: Leveraging Deep Depth Prediction for Monocular Direct Sparse Odometry (2018, background): “Self-supervised approaches [11,44,14] overcome this limitation by exploiting photoconsistency and geometric constraints to de-ﬁne loss functions, for example, in a stereo camera setup.”
Unsupervised Monocular Depth Estimation with Left-Right Consistency (2016, background): “Cs=αap(C l ap+C r ap)+αds(C l ds+C r ds)+αlr(C l lr+C r lr), (1) where Cap encourages the reconstructed image to appear similar to the corresponding training input, Cds enforces smooth disparities, and Clr prefers the predicted left and right disparities to be consistent.”