Competitive Collaboration: Joint Unsupervised Learning of Depth, Camera Motion, Optical Flow and Motion Segmentation

By Anurag Ranjan, Varun Jampani, Lukas Balles, Kihwan Kim, Deqing Sun, Jonas Wulff, Michael J. Black. 2018

640 citations recorded by the source corpus.

We address the unsupervised learning of several interconnected problems in low-level vision: single view depth prediction, camera motion estimation, optical flow, and segmentation of a video into the static scene and moving regions. Our key insight is that these four fundamental vision problems are coupled through geometric constraints. Consequently, learning to solve them together simplifies the problem because the solutions can reinforce each other. We go beyond previous work by exploiting geometry more explicitly and segmenting the scene into static and moving regions. To that end, we introduce Competitive Collaboration, a framework that facilitates the coordinated training of multiple specialized neural networks to solve complex problems. Competitive Collaboration works much like expectation-maximization, but with neural networks that act as both competitors to explain pixels that correspond to static or moving regions, and as collaborators through a moderator that assigns pixels to be either static or independently moving. Our novel method integrates all these problems in a common framework and simultaneously reasons about the segmentation of the scene into moving objects and the static background, the camera motion, depth of the static scene structure, and the optical flow of moving objects. Our model is trained without any supervision and achieves state-of-the-art performance among joint unsupervised methods on all sub-problems.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

On the Binding Problem in Artificial Neural Networks (2020, background): “motion) may play an important role in further improving these results (e.g. Pathak et al., 2017; Ranjan et al., 2019).”
Unsupervised Scale-consistent Depth and Ego-motion Learning from Monocular Video (2019, methodology): “2 shows the results of different network architectures and different resolution images, where "-VGG" stands for the method with DispNetS [5] and "-ResNet" stands for that with DispResNetS [9].”
Consistent video depth estimation (2020, methodology): “Recent methods improve the performance through incorporating coupled training with optical flow [Ranjan et al. 2019; Yin and Shi 2018; Zou et al. 2018], object motion [Dai et al.”
On the Uncertainty of Self-Supervised Monocular Depth Estimation (2020, background): “Seminal work in this direction is [82], extended by leveraging on point-cloud alignment [45], differentiable DVO [69], optical flow [78, 83, 11, 3], semantic [66] or scale consistency [5].”
Monocular depth estimation based on deep learning: An overview (2020, background): “[90] further extend the multi-task framework, and the motion segmentation is jointly trained with other tasks (depth, pose, flow) in an unsupervised way.”
Unifying Flow, Stereo and Depth Estimation (2022, result): “We also note that our unified model is different from multi-task joint training approaches [81], [82], [83], [84], [85].”
Every Pixel Counts ++: Joint Learning of Geometry and Motion with 3D Holistic Understanding (2018, methodology): “All the results are evaluated on the KITTI dataset using the corresponding standard metrics commonly used by other SoTA methods [6], [7], [18], [19].”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, methodology): “[49] 832× 256 CS + K 0.”
Self-Supervised Monocular Depth Hints (2019, background): “Depth regression is optimized and relative poses come from stereo camera calibration in a training-from-stereo setting [9, 7, 25, 27, 26], while depth values and camera poses can be optimized jointly when training on videos [42, 20, 37, 22, 38, 32, 44, 36, 28].”
DUSt3R: Geometric 3D Vision Made Easy (2023, positive): “Recent learning-based approaches formulate this problem either as pose and monocular depth regression [93, 172, 185] or pose and stereo matching [123, 131, 136, 142, 183].”
MonoViT: Self-Supervised Monocular Depth Estimation with a Vision Transformer (2022, background): “Other works, instead, directly tackle highly complex scenarios [67] and model rigid and non-rigid components present in the scene using the estimated depth, relative camera poses, and optical ﬂow in order to handle independent motions [47, 64, 72, 5, 39, 61] or by means of scene decomposition [41].”
Collaboration by Competition: Self-coordinated Knowledge Amalgamation for Multi-talent Student Learning (2020, methodology): “Some researchers [15] have recently introduced competitive collaboration mechanism to unsupervised multi task learning for some complex geometric coupled vision tasks.”
Digging Into Self-Supervised Monocular Depth Estimation (2018, methodology): “We also outperform recent methods ([38, 51]) that explicitly compute optical flow as well as motion masks.”