Unsupervised Learning of Depth and Ego-Motion from Video

By Tinghui Zhou, Matthew A. Brown, Noah Snavely, David G. Lowe. 2017

2,919 citations recorded by the source corpus.

We present an unsupervised learning framework for the task of monocular depth and camera motion estimation from unstructured video sequences. In common with recent work [10, 14, 16], we use an end-to-end learning approach with view synthesis as the supervisory signal. In contrast to the previous work, our method is completely unsupervised, requiring only monocular video sequences for training. Our method uses single-view depth and multiview pose networks, with a loss based on warping nearby views to the target using the computed depth and pose. The networks are thus coupled by the loss during training, but can be applied independently at test time. Empirical evaluation on the KITTI dataset demonstrates the effectiveness of our approach: 1) monocular depth performs comparably with supervised methods that use either ground-truth pose or depth for training, and 2) pose estimation performs favorably compared to established SLAM systems under comparable input settings.

Subfields: Aerospace Engineering, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Aerospace Engineering

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “For videos, self-supervision strategies include: leveraging temporal continuity via tracking [44, 45], predicting future [42], or preserving the equivariance of egomotion [13, 50, 30].”
Scaling and Benchmarking Self-Supervised Visual Representation Learning (2019, background): “Multi-modal information such as depth from a sensor [17], sound in a video [3, 4, 23, 53], sensors on an autonomous vehicle [2, 30, 79] etc.”
Shaping Belief States with Generative Environment Models for RL (2019, background): “A non-exhaustive list includes pixel control [55], contrastive predictive coding (CPC) [56], action-conditional CPC [20], frame reconstruction [35, 57], next-frame prediction [18, 58, 20, 59] and successor representations [60, 61, 62].”
Learning a Multi-View Stereo Machine (2017, background): “A line of recent works, complementary to ours, has proposed to incorporate ideas from multi-view geometry in a learning framework to train single view prediction systems [14, 56, 53, 42, 61] using multiple views as supervisory signal.”
Multi-view Supervision for Single-View Reconstruction via Differentiable Ray Consistency (2017, background): “[40] further relax the constraint of having calibrated stereo images, and learn a single-view depth model from monocular videos.”
Learning to Drive in a Day (2018, background): “Beyond pixel-space autoencoders is a wealth of computer vision research addressing effective compression of images: here existing work in areas such as semantic segmentation, depth, egomotion and pixel-flow provide an excellent prior for what is important in driving scenes [29, 1, 30].”
Transitive Invariance for Self-Supervised Visual Representation Learning (2017, background): “v:1708.02901v3 [cs.CV] 15 Aug 2017 carefully annotated data. However, large-scale, high-quality annotations come at a cost of expensive human effort. Unsupervised or “self-supervised” learning (e.g., [61,9, 45,63,64,35,44,62,40,66]) recently has attracted increasing interests because the “labels” ar”
Towards Robust Monocular Depth Estimation: Mixing Datasets for Zero-Shot Cross-Dataset Transfer (2019, background): “leverage self-supervision, but they either require stereo images [10], [23], [24] or exploit apparent motion [24], [25], [26], [27], and are thus difficult to apply to dynamic scenes.”
Boosting Few-Shot Visual Learning With Self-Supervision (2019, methodology): “Predicting the colors of images [31, 61], the relative position of image patches [6, 40], the random rotation that has been applied to an image [14], or the missing central part of an image [42], are some of the many methods [15, 32, 36, 56, 62] for self-supervised feature learning.”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, methodology): “We extract camera motion (rotation and translation) with an offthe-shelf camera pose estimator [52], and find YTVIS and UVO have a similar distributions of camera motions.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, background): “They can also be naturally integrated into deep learning frameworks through image reconstruction losses [70, 58, 66].”
GAN-Supervised Dense Visual Alignment (2021, background): “, classification) and applications such as robust filter learning [16, 36], view synthesis [26, 63, 90] and 3D representation learning [38, 83, 88].”
Deep Metric Transfer for Label Propagation with Limited Annotated Data (2018, background): “color [44], context [4], motion [46]) and thus may vary in performance on the target task we may want to transfer.”
Semantic Flow for Fast and Accurate Scene Parsing (2020, methodology): “A similar strategy is used in [63] for self-supervised monodepth learning via view synthesis.”
Self-Supervised Remote Sensing Feature Learning: Learning Paradigms, Challenges, and Future Works (2022, neutral): “[220], [221], [222], [223], [224], [225].”
Structured Attention Guided Convolutional Neural Fields for Monocular Depth Estimation (2018, neutral): “for jointly predicting depth and performing semantic segmentation, ego-motion estimation or surface normal computation [5, 38, 31].”
Who Let the Dogs Out? Modeling Dog Behavior from Visual Data (2018, background): “[54] propose an unsupervised approach for camera motion estimation.”
Deep Nets: What have They Ever Done for Vision? (2018, background): “supervision but only an energy function model (Ren et al., 2017; Zhou et al., 2017).”
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (2019, methodology): “2 shows the results of different network architectures and different resolution images, where "-VGG" stands for the method with DispNetS [5] and "-ResNet" stands for that with DispResNetS [9].”