Consistent video depth estimation

By Xuan Luo, Jia-Bin Huang, Richard Szeliski, Kevin Matzen, Johannes Peter Kopf. ACM Transactions on Graphics, 2020

419 citations recorded by the source corpus.

We present an algorithm for reconstructing dense, geometrically consistent depth for all pixels in a monocular video. We leverage a conventional structure-from-motion reconstruction to establish geometric constraints on pixels in the video. Unlike the ad-hoc priors in classical reconstruction, we use a learning-based prior, i.e., a convolutional neural network trained for single-image depth estimation. At test time, we fine-tune this network to satisfy the geometric constraints of a particular input video, while retaining its ability to synthesize plausible depth details in parts of the video that are less constrained. We show through quantitative validation that our method achieves higher accuracy and a higher degree of geometric consistency than previous monocular reconstruction methods. Visually, our results appear more stable. Our algorithm is able to handle challenging hand-held captured input videos with a moderate degree of dynamic motion. The improved quality of the reconstruction enables several applications, such as scene reconstruction and advanced video-based visual effects.

Subfields: Computer Vision and Pattern Recognition, Media Technology

View Paper: https://www.semanticscholar.org/paper/c09bc9f1fff9cb4731dc4f83924727fb3cc96e7e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Media Technology

Citations in the corpus

Neural Radiance Flow for 4D View Synthesis and Video Processing (2020, positive): “We further qualitatively evaluate our approach on monocular real world video from [38].”
A Comprehensive Survey on Test-Time Adaptation Under Distribution Shifts (2023, neutral): “[417], [418], [419], [420], ensuring temporal consistency between adjacent frames is a crucial aspect of the unsupervised learning objective.”
Dynamic View Synthesis from Dynamic Monocular Video (2021, background): “Recent work on depth estimation from monocular videos of dynamic scenes shows promising results [27, 58].”
Neural Scene Flow Fields for Space-Time View Synthesis of Dynamic Scenes (2020, methodology): “[40] estimates temporally consistent depth maps of scenes with small object motion by optimizing the weights of a single image depth prediction network, but we show that this approach fails to model large and complex 3D motions.”
NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video (2021, methodology): “Speciﬁcally, our method runs ∼ 10 × faster than Atlas, and 77 faster than Consistent Depth.”
NAS-DIP: Learning Deep Image Prior with Neural Architecture Search (2020, background): “Training CNNs on testing data has been shown as an effective approach for improving performance [55, 57, 76].”
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, background): “Early approaches to video depth estimation (Luo et al., 2020; Zhang et al., 2021) improve temporal consistency by fine-tuning depth models, and sometimes motion models, at test time for each input video.”
Depth-supervised NeRF: Fewer Views and Faster Training for Free (2021, background): “While these tools have become staples of 3D learning, recent works have focused on using deep learning to extract dense depth in the monocular setting [10, 23] or from videos [12].”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, methodology): “[38]; (“Inpainted”) its inpainted version, where disoccluded empty pixels in the 2D rendered images are inpainted using a recent video inpainting method [19]; and (“NeRF-T”) a version of NeRF with an extra time parameter, which is our model trained with only the color reconstruction loss (4).”
Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, positive): “In computer vision, the idea of learning at test time has been applied for decades to applications such as face detection [41], object detection [56], image super-resolution [68], and 3D reconstruction [53].”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “Knowing the depth to each pixel in an image has proved to be a useful and versatile tool, with applications ranging from augmented reality [59], autonomous driving [22], through to 3D reconstruction [64].”
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, methodology): “Previous work has investigated replacing hand-crafted with learned features[13, 7, 29, 26, 35], using neural 3D representations[46, 1, 9, 45, 44, 25, 22], and combining learned energy terms with classical optimization backends[58, 57].”