Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding

By Chenxu Luo, Zhenheng Yang, Peng Wang, Yang Wang, Wei Xu, Ramkant Nevatia, Alan L. Yuille. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018

313 citations recorded by the source corpus.

Learning to estimate 3D geometry in a single frame and optical flow from consecutive frames by watching unlabeled videos via deep convolutional network has made significant progress recently. Current state-of-the-art (SoTA) methods treat the two tasks independently. One typical assumption of the existing depth estimation methods is that the scenes contain no independent moving objects. while object moving could be easily modeled using optical flow. In this paper, we propose to address the two tasks as a whole, i.e., to jointly understand per-pixel 3D geometry and motion. This eliminates the need of static scene assumption and enforces the inherent geometrical consistency during the learning process, yielding significantly improved results for both tasks. We call our method as “Every Pixel Counts++” or “EPC++”. Specifically, during training, given two consecutive frames from a video, we adopt three parallel networks to predict the camera motion (MotionNet), dense depth map (DepthNet), and per-pixel optical flow between two frames (OptFlowNet) respectively. The three types of information, are fed into a holistic 3D motion parser (HMP), and per-pixel 3D motion of both rigid background and moving objects are disentangled and recovered. Various loss terms are formulated to jointly supervise the three networks. An effective adaptive training strategy is proposed to achieve better performance and more efficient convergence. Comprehensive experiments were conducted on datasets with different scenes, including driving scenario (KITTI 2012 and KITTI 2015 datasets), mixed outdoor/indoor scenes (Make3D) and synthetic animation (MPI Sintel dataset). Performance on the five tasks of depth estimation, optical flow estimation, odometry, moving object segmentation and scene flow estimation shows that our approach outperforms other SoTA methods, demonstrating the effectiveness of each module of our proposed method. Code will be available at: https://github.com/chenxuluo/EPC.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ba5c19a02696029566b9e01f5f55e908d25763bd

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Self-Supervised Monocular Depth Hints (2019, methodology): “Recently, multiple works [20, 28, 38, 44] have proposed to model the relationship of pixels in the consecutive frames of a video with joint estimation of optical flow, depth and camera poses with loss terms that supervise the different estimates to be consistent.”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “By simultaneously predicting optical flow and depth, existing works impose losses for cross-task consistency [35,37,60,63], geometric constraints [8,49], and modified reconstruction of the warped image [8,64], all approaches extending the image projection model to moving DC objects.”
Digging Into Self-Supervised Monocular Depth Estimation (2018, methodology): “We also outperform recent methods ([38, 51]) that explicitly compute optical flow as well as motion masks.”
On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “Finally, approaches leveraging both kind of supervisions have been proposed in [80, 77, 41, 20].”