Improving Semantic Segmentation via Decoupled Body and Edge Supervision

By Xiangtai Li, Xia Li, Li Zhang, Guangliang Cheng, Jianping Shi, Zhouchen Lin, Shaohua Tan, Yunhai Tong. European Conference on Computer Vision, 2020

308 citations recorded by the source corpus.

Existing semantic segmentation approaches either aim to improve the object's inner consistency by modeling the global context, or refine objects detail along their boundaries by multi-scale feature fusion. In this paper, a new paradigm for semantic segmentation is proposed. Our insight is that appealing performance of semantic segmentation requires \textit{explicitly} modeling the object \textit{body} and \textit{edge}, which correspond to the high and low frequency of the image. To do so, we first warp the image feature by learning a flow field to make the object part more consistent. The resulting body feature and the residual edge feature are further optimized under decoupled supervision by explicitly sampling different parts (body or edge) pixels. We show that the proposed framework with various baselines or backbone networks leads to better object inner consistency and object boundaries. Extensive experiments on four major road scene semantic segmentation benchmarks including \textit{Cityscapes}, \textit{CamVid}, \textit{KIITI} and \textit{BDD} show that our proposed approach establishes new state of the art while retaining high efficiency in inference. In particular, we achieve 83.7 mIoU \% on Cityscape with only fine-annotated data. Code and models are made available to foster any further research (\url{this https URL}).

Subfields: Civil and Structural Engineering, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/d77294dca8bed76205fc7f66af9e399c1b33c031

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Civil and Structural Engineering

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, background): “, the encoder) is arguably the most important model component [7, 27, 55, 58].”
Deep Interactive Thin Object Selection (2021, methodology): “In our case, while more advanced design such as [42, 12, 20] might also work, we purposely opt for simpler design as the main goal of this work is to demonstrate the effectiveness of context-edge disentangle-ment for thin object segmentation task.”
SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation (2022, background): “…types of decoders for different goals, including achieving multi-scale receptive ﬁelds [94, 7, 78], collecting multi-scale semantics [64, 80, 8], enlarging receptive ﬁeld [5, 4, 62], strengthening edge features [95, 2, 16, 42, 90], and capturing global context [19, 34, 89, 40, 23, 26, 91].”
BAPA-Net: Boundary Adaptation and Prototype Alignment for Cross-domain Semantic Segmentation (2021, background): “In addition, the fine-grained segmentation of category boundaries [2, 28] has become one of the most challenging difficulties in current semantic segmentation tasks.”
HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation (2022, unknown): “This concept was further improved in different aspects including increasing the receptive field while preserving spatial details [2, 3, 6, 48, 65, 74, 83], integrating context information [30,75,76,78,91], attention mechanisms [18,32,69,84], refining boundaries [15,36,77], and Transformer-based arch”
Scaling Wide Residual Networks for Panoptic Segmentation (2020, background): “Speciﬁcally, the Wide-ResNet-38 (WR-38) [92], reﬁned by several human-crafted networks with extensive experiments, has been the de facto network back-bone for semantic segmentation [63, 73, 105, 82, 52], and instance segmentation [55, 32] on Cityscapes leader-board [14].”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “Since it was proposed, numerous efforts have been devoted to improving FCN, by, for example, enlarging the receptive field [15, 16, 25, 124, 128, 135]; strengthening context cues [4, 43, 47, 48, 56, 57, 70, 75, 77, 81, 90, 126, 128, 129, 132, 138, 141]; leveraging boundary information [6, 14, 27, 66”
Transformer-Based Visual Segmentation: A Survey (2023, background): “…including better encoder-decoder frameworks [58], [59], larger kernels [60], [61], multiscale pooling [11], [62], multiscale feature fusion [12], [63], [64], [65], non-local modeling [18], [66], [67], efﬁcient modeling [68], [69], [70], and better boundary delineation [71], [72], [73], [74].”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, methodology): “Additional FCN-based methods improve semantic segmentation by refining context priors [35], [36], [37], enhancing boundary cues [38], [39], [40], and appending various attention modules [41], [42], [43].”