Depth Extraction from Video Using Non-parametric Sampling

By Kevin Karsch, Ce Liu, Sing Bing Kang. 2012

151 citations recorded by the source corpus.

We describe a technique that automatically generates plausible depth maps from videos using non-parametric depth sampling. We demonstrate our technique in cases where past methods fail (non-translating cameras and dynamic scenes). Our technique is applicable to single images as well as videos. For videos, we use local motion cues to improve the inferred depth maps, while optical flow is used to ensure temporal depth consistency. For training and evaluation, we use a Kinect-based system to collect a large dataset containing stereoscopic videos with known depths. We show that our depth estimation technique outperforms the state-of-the-art on benchmark databases. Our technique can be used to automatically convert a monoscopic video into stereo for 3D visualization, and we demonstrate this through a variety of visually pleasing results for indoor and outdoor scenes, including results from the feature film Charade.

Subfields: Computer Vision and Pattern Recognition, Media Technology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Depth Map Prediction from a Single Image using a Multi-Scale Deep Network (2014, methodology): “[7] use a kNN transfer mechanism based on SIFT Flow [11] to estimate depths of static backgrounds from single images, which they augment with motion information to better estimate moving foreground subjects in videos.”
Deeper Depth Prediction with Fully Convolutional Residual Networks (2016, methodology): “A second cluster of related work comprises nonparametric approaches for depth transfer [10, 13, 18, 20], which typically perform feature-based matching (e.”
Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos (2018, result): “These approaches have shown compelling re-sults as compared to traditional methods (Karsch, Liu, and Kang 2014b).”
Discrete-Continuous Depth Estimation from a Single Image (2014, methodology): “Method rel log10 rms Depth transfer [13] C1 0.”
Predicting Depth, Surface Normals and Semantic Labels with a Common Multi-scale Convolutional Architecture (2014, background): “Ladicky[20]Karsch[18] Baig [1] Liu [23] Eigen[8] Ours(A) Ours(VGG)”
DeepLiDAR: Deep Surface Normal Guided Depth Prediction for Outdoor Scene From Sparse LiDAR Data and Single Color Image (2018, methodology): “Early methods mainly relied on the hand-crafted features and probabilistic graphical models [42, 21, 22, 24, 31].”
Deformable Spatial Pyramid Matching for Fast Dense Correspondences (2013, background): “This advance beyond instance matching leads to many interesting new applications, such as semantic image segmentation [15], image completion [2], image classification [11], and video depth estimation [10].”
Sparse-to-Dense: Depth Prediction from Sparse Depth Samples and a Single Image (2017, methodology): “Non-parametric approaches [9–12] were also exploited to estimate the depth of a query image by combining the depths of images with similar photometric content retrieved from a database.”
FastDepth: Fast Monocular Depth Estimation on Embedded Systems (2019, methodology): “Non-parametric approaches [7–10] were also exploited to estimate the depth of a query image by combining the depths of images with similar photometric content retrieved from a database.”