Real-Time Self-Adaptive Deep Stereo

By Alessio Tonioni, Fabio Tosi, Matteo Poggi, Stefano Mattoccia, Luigi Di Stefano. Computer Vision and Pattern Recognition, 2018

296 citations recorded by the source corpus.

Deep convolutional neural networks trained end-to-end are the state-of-the-art methods to regress dense disparity maps from stereo pairs. These models, however, suffer from a notable decrease in accuracy when exposed to scenarios significantly different from the training set (e.g., real vs synthetic images, etc.). We argue that it is extremely unlikely to gather enough samples to achieve effective training/tuning in any target domain, thus making this setup impractical for many applications. Instead, we propose to perform unsupervised and continuous online adaptation of a deep stereo network, which allows for preserving its accuracy in any environment. However, this strategy is extremely computationally demanding and thus prevents real-time inference. We address this issue introducing a new lightweight, yet effective, deep stereo architecture, Modularly ADaptive Network(MADNet), and developing a Modular ADaptation (MAD) algorithm, which independently trains sub-portions of the network. By deploying MADNet together with MAD we introduce the first real-time self-adaptive deep stereo system enabling competitive performance on heterogeneous datasets. Our code is publicly available at https://github.com/CVLAB-Unibo/Real-time-self-adaptive-deep-stereo.

Subfields: Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/98333bedec446377cac57d63140ce0d4e79c097d

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, methodology): “Inspired by the previous coarse-to-fine learning-based stereo approaches [8, 9, 11], we present a novel cascade formulation of 3D cost volumes.”
Continual Learning for Real-World Autonomous Systems: Algorithms, Challenges and Frameworks (2021, background): “with high prediction error that is D3 [77].”
AANet: Adaptive Aggregation Network for Efficient Stereo Matching (2020, methodology): “2D methods [20, 18, 33] generally adopt a correlation layer [20] while 3D methods [14, 4, 25, 43, 3] mostly use direct feature concatenation [14].”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “Zhong et al. [142] and Tonioni et al. [84] consider online adaptation separately from the initial training.”
DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (2019, methodology): “Comparing to previous approaches on KITTI, DeepPruner achieves competitive performance to state-of-the-art methods [8, 33] and ranks first among all real-time models [23, 32].”
CFNet: Cascade and Fused Cost Volume for Robust Stereo Matching (2021, methodology): “Many previous work, including Dispnet [18], MADNet [29], and AANet [32], belong to this category.”