Depth Transfer: Depth Extraction from Video Using Non-Parametric Sampling

By Kevin Karsch, Ce Liu, Sing Bing Kang. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2014

617 citations recorded by the source corpus.

We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large data set containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/5f8f3e0fa482cf38a77af3af08b0375f1c49415a

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “Early work on monocular depth estimation used MRF-based formulations [3], simple geometric assumptions [2], or nonparametric methods [14].”
Unsupervised Monocular Depth Estimation with Left-Right Consistency (2016, methodology): “[28] attempt to produce more consistent image level predictions by copying whole depth images from a training set.”
GeoNet: Geometric Neural Network for Joint Depth and Surface Normal Estimation (2018, neutral): “DepthTransfer [13] 1.”
Deep Stereo: Learning to Predict New Views from the World's Imagery (2015, methodology): “r scans as training data, and the automatic photo pop-up work of Hoiem et al. [14], which uses images with manually annotated geometric classes. More recent methods have used Kinect data for training [15,18] and deep learning methods for single view depth or surface normal prediction [9,33]. However”
Enforcing Geometric Constraints of Virtual Normal for Depth Prediction (2019, neutral): “[23] 0.”
Temporal Tessellation: A Unified Approach for Video Analysis (2016, methodology): “ich provided a robust means for computing and using dense correspondences. Brieﬂy, these methods assume that reference images are associated with a semantic channel containing, e.g., per pixel depths [13, 18], semantic segmentation labels [24, 42] and even character labels for text images [15]. Robu”
Depth and surface normal estimation from monocular images using regression on deep features and hierarchical CRFs (2015, methodology): “Method rel log10 rms Depth transfer [13] C1 0.”
Pseudo-LiDAR From Visual Depth Estimation: Bridging the Gap in 3D Object Detection for Autonomous Driving (2018, background): “The accuracy of these systems has increased dramatically since early work on monocular depth estimation [8, 15, 26].”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “Early methods mainly relied on the hand-crafted features and probabilistic graphical models [42, 21, 22, 24, 31].”
Deep convolutional neural fields for depth estimation from a single image (2014, methodology): “As can be observed, our method outperforms classic methods like Make3d [15], DepthTransfer [5] with large margins.”
Face anti-spoofing using patch and depth-based CNNs (2017, neutral): “In recent years there have been a rapid progress of data-driven methods [23], especially deep neural networks trained on large RGB-D datasets [40], as well as weak annotations [10].”
Monocular Object Instance Segmentation and Depth Ordering with CNNs (2015, methodology): “The depth map for each image is computed using the depthfrom-single-image method of [18].”
Learning Depth from Monocular Videos Using Direct Methods (2017, neutral): “[14] depth 0.”
Dual Attention Guided Gaze Target Detection in the Wild (2021, background): “Early works [19, 21, 23] make an effort to exploit some statistically meaningful monocular cues (e.g., perspective, object sizes, and object localization).”
Consistent video depth estimation (2020, methodology): “…several efforts have been made to make the estimated depth more temporally consistent by explicitly applying optical flow-based consistency loss [Karsch et al. 2014] or implicitly encouraging temporal consistency using recurrent neural networks [Patil et al. 2020; Wang et al. 2019b; Zhang et…”
Monocular Fisheye Camera Depth Estimation Using Sparse LiDAR Supervision (2018, methodology): “Through example images [20], [21] it is found that the idea of depth transfer can be used to predict depth map or integrate depth map prediction with semantic segmentation [1], [10], [22] in supervised training.”
Multi-scale Continuous CRFs as Sequential Deep Networks for Monocular Depth Estimation (2017, background): “Karsch et al. [14] developed Depth Transfer, a non parametric approach where the depth of an input image is reconstructed by transferring the depth of multiple similar images and then applying some warping and optimizing procedures.”
Data-Driven 3D Primitives for Single Image Understanding (2013, methodology): “[17]: One can also produce surface normals by predicting depth and computing normals on the results; we do this with the depth prediction method of Karsch et al.”