GeoNet: Unsupervised Learning of Dense Depth, Optical Flow and Camera Pose

By Zhichao Yin, Jianping Shi. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

1,275 citations recorded by the source corpus.

We propose GeoNet, a jointly unsupervised learning framework for monocular depth, optical flow and egomotion estimation from videos. The three components are coupled by the nature of 3D scene geometry, jointly learned by our framework in an end-to-end manner. Specifically, geometric relationships are extracted over the predictions of individual modules and then combined as an image reconstruction loss, reasoning about static and dynamic scene parts separately. Furthermore, we propose an adaptive geometric consistency loss to increase robustness towards outliers and non-Lambertian regions, which resolves occlusions and texture ambiguities effectively. Experimentation on the KITTI driving dataset reveals that our scheme achieves state-of-the-art results in all of the three tasks, performing better than previously unsupervised methods and comparably with supervised ones.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/4e1b1b98d9b0c73eaca4e71f553e72503fbd4aba

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Lite-Mono: A Lightweight CNN and Transformer Architecture for Self-Supervised Monocular Depth Estimation (2022, methodology): “By replacing the network architecture from the VGG model [33] with a ResNet [17] Yin et al. [41] achieved better results.”
Image Matching from Handcrafted to Deep Features: A Survey (2020, background): “2018) and camera localization or transformation estimation (Poursaeed et al. 2018; Erlik Nowruzi et al. 2017; Yin and Shi 2018).”
Monocular depth estimation based on deep learning: An overview (2020, methodology): “[15], and Wang et al.”
Omnidata: A Scalable Pipeline for Making Multi-Task Mid-Level Vision Datasets from 3D Scans (2021, background): “For example, the indoor scene dataset NYU [53], still used for training some state-of-the-art depth estimation models [70], contains only 795 training images—all taken with a single camera.”
Depth-Aware Video Frame Interpolation (2019, background): “We note that several approaches jointly estimate optical flow and depth by exploiting the cross-task constraints and consistency [40, 42, 43].”
Depth From Videos in the Wild: Unsupervised Monocular Depth Learning From Unknown Cameras (2019, methodology): “Techniques were developed for the monocular [33, 42, 24, 41, 44, 35, 6] and binocular [12, 33, 40, 46, 35, 46] settings.”
BARF: Bundle-Adjusting Neural Radiance Fields (2021, background): “They can also be naturally integrated into deep learning frameworks through image reconstruction losses [70, 58, 66].”
Depth Prediction Without the Sensors: Leveraging Structure for Unsupervised Learning from Monocular Videos (2018, methodology): “Many subsequent works have improved the initial results in both the monocular setting (Yang et al. 2017; Yin 2018) and when using stereo during training (Godard, Aodha, and Brostow 2017; Ummen-hofer et al. 2017; Zhan et al. 2018; Yang et al. 2018a).”
Self-Supervised Monocular Depth Hints (2019, background): “Depth regression is optimized and relative poses come from stereo camera calibration in a training-from-stereo setting [9, 7, 25, 27, 26], while depth values and camera poses can be optimized jointly when training on videos [42, 20, 37, 22, 38, 32, 44, 36, 28].”
Consistent video depth estimation (2020, methodology): “Recent methods improve the performance through incorporating coupled training with optical flow [Ranjan et al. 2019; Yin and Shi 2018; Zou et al. 2018], object motion [Dai et al. 2019; Vijayanarasimhan et al. 2017], surface normal [Qi et al. 2018], edge [Yang et al. 2018], and visual odometry…”
DF-Net: Unsupervised Joint Learning of Depth and Flow using Cross-Task Consistency (2018, methodology): “The synthesized flow from depth and pose can either be used for flow prediction in rigid regions [60,65,68,48] as is or used for view synthesis to train depth model using monocular videos [73].”
SurroundOcc: Multi-Camera 3D Occupancy Prediction for Autonomous Driving (2023, methodology): “While early methods require full depth annotations to supervise the depth estimation model [14, 24, 67], later research focuses on self-supervised depth estimation as it does not require intensive human annotations [71, 16, 4, 68, 63, 57, 70, 44, 8].”