SegStereo: Exploiting Semantic Information for Disparity Estimation

By Guorun Yang, Hengshuang Zhao, Jianping Shi, Zhidong Deng, Jiaya Jia. European Conference on Computer Vision, 2018

377 citations recorded by the source corpus.

Disparity estimation for binocular stereo images finds a wide range of applications. Traditional algorithms may fail on featureless regions, which could be handled by high-level clues such as semantic segments. In this paper, we suggest that appropriate incorporation of semantic cues can greatly rectify prediction in commonly-used disparity estimation frameworks. Our method conducts semantic feature embedding and regularizes semantic cues as the loss term to improve learning disparity. Our unified model SegStereo employs semantic features from segmentation and introduces semantic softmax loss, which helps improve the prediction accuracy of disparity maps. The semantic cues work well in both unsupervised and supervised manners. SegStereo achieves state-of-the-art results on KITTI Stereo benchmark and produces decent prediction on both CityScapes and FlyingThings3D datasets.

Subfields: Computer Vision and Pattern Recognition, Environmental Engineering

View Paper: https://www.semanticscholar.org/paper/92de2ed3805968d6d95da4fa9c44423ef50a6a37

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Environmental Engineering

Citations in the corpus

Unknown Object Segmentation from Stereo Images (2021, methodology): “After each, a correlation layer [46], [47] restricted to a local horizontal region to capture stereoscopic information is applied.”
Group-Wise Correlation Stereo Network (2019, methodology): “There are also works integrating additional information such as edge features [22] and semantic features [26].”
KeyPose: Multi-View 3D Labeling and Keypoint Estimation for Transparent Objects (2019, methodology): “The success of deep-learning methods in computer vision inspired research in this area, using end-to-end deep networks equipped with a correlation cost volume [17, 9, 6, 32], or point-based depth representation and iterative refinement [4].”
Semantic Stereo for Incidental Satellite Images (2018, background): “The two subtasks may be considered separately or sequentially and then fused [26, 27], semantic segmentation may be utilized as a prior for stereo [25, 28], stereo may be utilized as a prior for semantic segmentation [33], or both may be solved together [3, 29].”
A Survey on Deep Learning Techniques for Stereo-Based Depth Estimation (2020, methodology): “The magnitude of the ﬁrst-order gradient ∇ of the estimated disparity map [68]: • The magnitude of the second-order gradient of the estimated disparity [127], [132]: • The second-order gradient of the estimated disparity map weighted by the image’s second-order gradients [70]: (2) Consistency.”
Self-Supervised Monocular Depth Estimation: Solving the Dynamic Object Problem by Semantic Guidance (2020, background): “s. Multi-Task Learning: Multi-task learning has shown improvements in many research elds, e.g., domain adaptation [3,41,69], depth estimation [11,47,68] and semantic segmentation [28,30]. Yang et al. [62] incorporated a semantic segmentation cue into the self-supervised depth estimation with stereo ”
AutoDispNet: Improving Disparity Estimation With AutoML (2019, neutral): “87% SegStereo [72] - - 1.”
DeepPruner: Learning Efficient Stereo Matching via Differentiable PatchMatch (2019, background): “With this in mind, researchers propose to directly regress sub-pixel disparities from the given stereo images [35, 9, 29].”
Hierarchical Deep Stereo Matching on High-Resolution Images (2019, background): “Recent “deep” stereo methods perform well on low-resolution benchmarks [5, 11, 16, 21, 38], while failing to produce SOTA results on high-res benchmarks [26].”
Cascade Cost Volume for High-Resolution Multi-View Stereo and Stereo Matching (2019, methodology): “Typically, these methods downsample the feature maps to formulate the cost volumes at a lower resolution [3,4,8,12,15,22,26,38,44,45,48], and adopt upsampling [3, 8, 15, 26, 34, 38, 41, 48] or postrefinement [4, 22] to output the final high-resolution result.”
DSNet: Joint Learning for Scene Segmentation and Disparity Estimation (2019, background): “…77.3 TuSimple [34] 77.6 SAC-multiple [42] 78.1 PSPNet [43] 78.4 DeepLabv3+ [5] 79.6 3.43 PBCP [30] 3.61 MC-CNN-acrt [39] 3.89 chen17 [7] 4.07 PRSM [33] 4.27 DispNetC [26] 4.34 Content-CNN [25] 4.54 SPS-St [37] 5.31 OSF [27] 5.79 MDP [20] 6.74 robust features, which widely used in [28] [31] [38].”
Iterative Geometry Encoding Volume for Stereo Matching (2023, methodology): “Cost Filtering-based Methods To improve the representative ability of a cost volume, most existing learning-based stereo methods [5, 12, 23, 33, 45, 46, 49, 52] construct a cost volume using powerful CNN features.”
Kimera: From SLAM to spatial perception with 3D dynamic scene graphs (2021, background): “…the 2D semantic labels can be obtained using off-the-shelf tools for pixel-level 2D semantic segmentation, e.g., deep neural networks (Lang et al. 2019; Zhang et al. 2019a; Chen et al. 2017; Zhao et al. 2017; Yang et al. 2018; Paszke et al. 2016; Ren et al. 2015; He et al. 2017; Hu et al. 2017).”
Improving Semantic Segmentation via Video Propagation and Label Relaxation (2018, neutral): “SegStereo [40] 59.”