Dynamic-Structured Semantic Propagation Network

By Xiaodan Liang, Hongfei Zhou, Eric P. Xing. 2018

166 citations recorded by the source corpus.

Semantic concept hierarchy is still under-explored for semantic segmentation due to the inefficiency and complicated optimization of incorporating structural inference into dense prediction. This lack of modeling semantic correlations also makes prior works must tune highly-specified models for each task due to the label discrepancy across datasets. It severely limits the generalization capability of segmentation models for open set concept vocabulary and annotation utilization. In this paper, we propose a Dynamic-Structured Semantic Propagation Network (DSSPN) that builds a semantic neuron graph by explicitly incorporating the semantic concept hierarchy into network construction. Each neuron represents the instantiated module for recognizing a specific type of entity such as a super-class (e.g. food) or a specific concept (e.g. pizza). During training, DSSPN performs the dynamic-structured neuron computation graph by only activating a sub-graph of neurons for each image in a principled way. A dense semantic-enhanced neural block is proposed to propagate the learned knowledge of all ancestor neurons into each fine-grained child neuron for feature evolving. Another merit of such semantic explainable structure is the ability of learning a unified model concurrently on diverse datasets by selectively activating different neuron sub-graphs for each annotation at each step. Extensive experiments on four public semantic segmentation datasets (i.e. ADE20K, COCO-Stuff, Cityscape and Mapillary) demonstrate the superiority of our DSSPN over state-of-the-art segmentation models. Moreoever, we demonstrate a universal segmentation model that is jointly trained on diverse datasets can surpass the performance of the common fine-tuning scheme for exploiting multiple domain knowledge.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “Among the approaches, most of methods [41, 42, 43, 20, 34, 40] adopt the ResNet-101 as backbone and RefineNet [21] adopts a more powerful network, i.”
Deep High-Resolution Representation Learning for Visual Recognition (2019, neutral): “DSSPN [88] D-ResNet-101 76.”
OCNet: Object Context Network for Scene Parsing (2018, background): “101 78:1 DUC-HDC [23]z WACV2018 ResNet-101 77:6 AAF [9]z ECCV2018 ResNet-101 79:1 BiSeNet [28]z ECCV2018 ResNet-101 78:9 PSANet [34]z ECCV2018 ResNet-101 80:1 DFN [29]z CVPR2018 ResNet-101 79:3 DSSPN [12]z CVPR2018 ResNet-101 77:8 DepthSeg [10]z CVPR2018 ResNet-101 78:2 DenseASPP [27]z CVPR2018 Dens”
Image Segmentation Using Deep Learning: A Survey (2020, background): “Additional models and methods include BoxSup [119], Graph Convolutional Networks (GCN) [120], Wide ResNet [121], Exfuse [122] (enhancing low-level and high-level features fusion), Feedforward-Net [123], saliency-aware models for geodesic video segmentation [124], Dual Image Segmentation (DIS) [125],”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2020, methodology): “Liang et al. [18] train a model by mixing Cityscapes, ADE20K, COCO Stuff, and Mapillary, but do not evaluate cross-dataset generalization.”
High-Resolution Representations for Labeling Pixels and Regions (2019, unknown): “DSSPN [59] Dilated-ResNet-101 76.”
CIRL: Controllable Imitative Reinforcement Learning for Vision-based Self-driving (2018, background): “Although visual perception tasks have been well studied by resorting to supervised learning on large-scale datasets [4,5], simplistic driving policies by manually designed rules in the modular pipeline is far from sufﬁcient for handling diverse real-world cases as discussed in [6,7].”
Survey on semantic segmentation using deep learning techniques (2019, background): “[43] proposed a Dynamic Structured Semantic Propagation Network (DSSPN), that builds a large semantic neuron graph by taking in the semantic concept hierarchy into network construc-”
Adaptive Pyramid Context Network for Semantic Segmentation (2019, unknown): “…32.31 CascadeNet [42] 34.90 ReﬁneNet [17] ResNet152 40.7 PSPNet [40] ResNet101 43.29 PSPNet [40] ResNet269 44.94 EncNet [38] ResNet101 44.65 SAC [39] ResNet101 44.30 PSANet [13] ResNet101 43.77 UperNet [33] ResNet101 42.66 DSSPN [15] ResNet101 43.68 OCNet [37] ResNet101 45.08 Ours ResNet101 45.38”
MSeg: A Composite Dataset for Multi-Domain Semantic Segmentation (2022, methodology): “[18] train a model by mixing Cityscapes, ADE20K, COCO Stuff, and Mapillary, but do not evaluate cross-dataset generalization.”
Adaptive Context Network for Scene Parsing (2019, unknown): “Res-101 ReﬁneNet [22] 33.6 Ding et al.[7] 35.7 DSSPN[20] 38.9 SGR [19] 39.1 DANet[16] 39.7 ACNet 40.1”
Symbolic Graph Reasoning Meets Convolutions (2018, methodology): “The very recent DSSPN [27] directly designs a network layer for each parent concept.”