DeMoN: Depth and Motion Network for Learning Monocular Stereo

By Benjamin Ummenhofer, Huizhong Zhou, Jonas Uhrig, Nikolaus Mayer, Eddy Ilg, Alexey Dosovitskiy, Thomas Brox. Computer Vision and Pattern Recognition, 2016

748 citations recorded by the source corpus.

In this paper we formulate structure from motion as a learning problem. We train a convolutional network end-to-end to compute depth and camera motion from successive, unconstrained image pairs. The architecture is composed of multiple stacked encoder-decoder networks, the core part being an iterative network that is able to improve its own predictions. The network estimates not only depth and motion, but additionally surface normals, optical flow between the images and confidence of the matching. A crucial component of the approach is a training loss based on spatial relative differences. Compared to traditional two-frame structure from motion methods, results are more accurate and more robust. In contrast to the popular depth-from-single-image networks, DeMoN learns the concept of matching and, thus, better generalizes to structures not seen during training.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/84624c837102997784ce630b392d0975b642a4db

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

DeepMVS: Learning Multi-view Stereopsis (2018, methodology): “We compare our approach with DeMoN [37] because it is the closest to ours among the existing learning-based stereopsis methods.”
Learning to Find Good Correspondences (2017, methodology): “For SUN3D we choose 9 sequences for training, and use the 15 sequences previously used by [31], collectively marked with ‡, only for testing.”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, methodology): “We could explore other well-designed smoothness terms like edge aware loss [47] or scale invariant gradient loss [39] to further improve”
Revisiting Single Image Depth Estimation: Toward Higher Resolution Maps With Accurate Object Boundaries (2018, methodology): “There are other works that use gradient-based losses including DeMon [36], which consider different problems from single image depth estimation.”
Single-View View Synthesis With Multiplane Images (2020, background): “To address this ambiguity, prior work on single-view depth prediction typically employs a scale-invariant depth loss [3, 32] or more recently even a scale-and-shift invariant depth loss [15].”
DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras (2021, methodology): “These methods are not full SLAM systems, but instead focus on small scale reconstruction on the order of two [8, 50, 53] up to a dozen frames [59, 46, 47].”
CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAM (2018, background): “Some particularly eye-catching pieces of work over the past year have focused on supervised and self-supervised training of surprisingly capable networks which are able to estimate visual odometry, depth and other quantities from video [10, 29, 30, 4, 32, 31].”
The Temporal Opportunist: Self-Supervised Multi-Frame Monocular Depth (2021, methodology): “Some methods can predict pose at test time, but they still need to be trained with supervision e.g. [77].”
NICE-SLAM: Neural Implicit Scalable Encoding for SLAM (2021, background): “For example, [54, 68] regress both depth and pose updates.”
NeurAll: Towards a Unified Model for Visual Perception in Automated Driving (2019, background): “CNNs have gradually shown high accuracy for estimation of dense optical ﬂow [13] and of monocular depth [14].”
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, background): “Unsupervised learning of depth, where the only supervision is obtained from the monocular video itself and no depth sensors are needed, has also been popularized recently [47, 10, 12, 33, 24, 34, 44].”
Deep Stereo Using Adaptive Thin Volume Representation With Uncertainty Awareness (2019, background): “A depth map represents dense 3D information that is perfectly aligned with a reference view; depth reconstruction has been demonstrated in many previous works on reconstruction with both single view [9, 51, 16, 17, 62] and multiple views [4, 48, 18, 14, 43, 57, 43].”
VGGT: Visual Geometry Grounded Transformer (2025, neutral): “Recent methods [ 3, 67, 71, 74, 75, 78, 81, 83, 89, 108] explored end-to-end differentiable SfM, where VG-GSfM [83] started to outperform traditional algorithms on challenging phototourism scenarios.”
BA-Net: Dense Bundle Adjustment Network (2018, methodology): “In Ummenhofer et al. (2017), the scene depth and the camera motion are predicted from optical ﬂow features, which help to make it generalizing better to unseen data.”
Visual SLAM for Automated Driving: Exploring the Applications of Deep Learning (2018, neutral): “When the depth estimation is supervised, the loss function usually reads as regression loss [18, 39, 41, 59] w/wo regularization terms [70].”
Detecting Photoshopped Faces by Scripting Photoshop (2019, methodology): “Following [34], we encourage the flow to be smooth by minimizing a multiscale loss on the flow gradients:”
NeuralRecon: Real-Time Coherent 3D Reconstruction from Monocular Video (2021, methodology): “25 ↑ Abs Rel ↓ Sq Rel ↓ RMSE ↓ Time ↓ DeMoN [45] 31.”
GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose (2018, background): “For example, they require large quantities of laser scanned depth data for supervision [48], demand stereo cameras as additional equipment for data acquisition [15], or cannot explicitly handle non-rigidity and occlusions [50, 56].”