DeepV2D: Video to Depth with Differentiable Structure from Motion

By Zachary Teed, Jia Deng. International Conference on Learning Representations, 2018

325 citations recorded by the source corpus.

We propose DeepV2D, an end-to-end deep learning architecture for predicting depth from video. DeepV2D combines the representation ability of neural networks with the geometric principles governing image formation. We compose a collection of classical geometric algorithms, which are converted into trainable modules and combined into an end-to-end differentiable architecture. DeepV2D interleaves two stages: motion estimation and depth estimation. During inference, motion and depth estimation are alternated and converge to accurate depth. Code is available this https URL.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/dfb22535ada5e35b9ed3523499127003e4afc00d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video (2021, methodology): “Method Comp ↓ Acc ↓ Recall ↑ Prec ↑ F-score ↑ DeepV2D [44] 0.”
Depth Anything 3: Recovering the Visual Space from Any Views (2025, unknown): “Early end-to-end approaches [86, 90] moved beyond modular SfM/MVS pipelines by directly regressing camera poses and per-image depths from pairs of images.”
Consistent video depth estimation (2020, methodology): “ical consistency of the reconstruction over time either implicitly via recurrent neural networks [Patil et al. 2020; Wang et al. 2019b] or explicitly using multi-view reconstruction [Liu et al. 2019; Teed and Deng 2020]. State-of-the-art video-based depth estimation methods [Liu et al. 2019; Teed an”
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion (2024, background): “Given a set of 2D images, structure from motion (SfM) (Sch¨onberger & Frahm, 2016; Teed & Deng, 2018; Tang & Tan, 2018) or visual SLAM (Teed & Deng, 2021; Mur-Artal et al., 2015; Mur-Artal & Tard´os, 2017; Engel et al., 2014; Newcombe et al., 2011) estimate 3D structure of a scene while also…”
VGGT: Visual Geometry Grounded Transformer (2025, unknown): “Recent methods [ 3, 67, 71, 74, 75, 78, 81, 83, 89, 108] explored end-to-end differentiable SfM, where VG-GSfM [83] started to outperform traditional algorithms on challenging phototourism scenarios.”
Unifying Flow, Stereo and Depth Estimation (2022, methodology): “However, like rectiﬁed stereo matching, the state-of-the-art methods are usually dominated by 3D convolution-based approaches [72], [73], which accordingly introduces cubic computational complexity.”
DUSt3R: Geometric 3D Vision Made Easy (2023, unknown): “They are essentially based on the idea of building a differentiable SfM pipeline, replicating the traditional pipeline but training it end-to-end [131, 136, 184].”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “DeepV2D [51] similarly alternates between regressing depth and pose estimation but uses test-time optimization.”
RAFT: Recurrent All-Pairs Field Transforms for Optical Flow (2020, methodology): “This has motivated several works to embed optimization problems into network architectures [4,3,38,28,39].”
Space-time Neural Irradiance Fields for Free-Viewpoint Video (2020, methodology): “Several monocular video depth estimation methods predict depth using the cost volume computed from nearby frames [66, 33].”
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, methodology): “These methods are not full SLAM systems, but instead focus on small scale reconstruction on the order of two [8, 50, 53] up to a dozen frames [59, 46, 47].”