Improving Semantic Segmentation via Video Propagation and Label Relaxation

By Yi Zhu, Karan Sapra, Fitsum A. Reda, Kevin J. Shih, Shawn D. Newsam, Andrew Tao, Bryan Catanzaro. 2018

415 citations recorded by the source corpus.

Semantic segmentation requires large amounts of pixel-wise annotations to learn accurate models. In this paper, we present a video prediction-based methodology to scale up training sets by synthesizing new training samples in order to improve the accuracy of semantic segmentation networks. We exploit video prediction models' ability to predict future frames in order to also predict future labels. A joint propagation strategy is also proposed to alleviate mis-alignments in synthesized samples. We demonstrate that training segmentation models on datasets augmented by the synthesized samples leads to significant improvements in accuracy. Furthermore, we introduce a novel boundary label relaxation technique that makes training robust to annotation noise and propagation artifacts along object boundaries. Our proposed methods achieve state-of-the-art mIoUs of 83.5% on Cityscapes and 82.9% on CamVid. Our single model, without model ensembles, achieves 72.8% mIoU on the KITTI semantic segmentation test set, which surpasses the winning entry of the ROB challenge 2018.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, neutral): “[101] C, V, MV - - 83.”
Object-Contextual Representations for Semantic Segmentation (2019, neutral): “10 VPLR†‡ [82] WideResNet-38 + ASPP 4× M - 83.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, neutral): “[17] C, V, MV - - 83.”
ResNeSt: Split-Attention Networks (2020, methodology): “r vision research. Networks trained for image classication often serve as the backbone of the neural networks designed for other applications, such as object detection [22,46], semantic segmentation [6,43,73] and pose estimation [14,58]. Recent work has signi- cantly boosted image classication accur”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, methodology): “Most recent semantic segmentation work for Cityscapes in particular has utilized the ~20, 000 coarsely labelled images as-is for training state-of-the-art models [12, 29].”
A Survey on Deep Learning Technique for Video Segmentation (2021, methodology): “[222], [234] use both labeled and unlabeled video frames to learn VSS.”
Improving Semantic Segmentation via Decoupled Body and Edge Supervision (2020, methodology): “iding-window manner for a fair comparison. 4.1 Ablation studies Improvements over baseline model. We rst apply our method on naive dilated FCN models [6], where we also include uniform sampling trick [23] to balance classes during training as our strong baselines in Table 1(a). Our naive FCNs achiev”
DACS: Domain Adaptation via Cross-domain Mixed Sampling (2020, background): “Many pixels of low conﬁdence are aligned with predictions at semantic boundaries [24, 25, 9], thus leading to a diminished training signal there.”
MonoScene: Monocular 3D Semantic Scene Completion (2021, methodology): “For training only, JS3CNetrgb also requires a semantic point cloud (x̂sem pts), obtained by augmenting x̂pts with 2D semantic labels from a pretrained network [82].”
Semantic Flow for Fast and Accurate Scene Parsing (2020, methodology): “ore detailed information about Mapillary pretraining and TensorRT acceleration can be referred in supplementary ﬁle. Comparison with state-of-the-art accurate models: State-of-the-art accurate models [13, 49, 60, 65] perform multi-scale and horizontal ﬂip inference to achieve better results on the C”
Detection and Retrieval of Out-of-Distribution Objects in Semantic Segmentation (2020, methodology): “dataset are taken from the GitHub repository of [38].”
Image Classification with Deep Learning in the Presence of Noisy Labels: A Survey (2019, background): “As mentioned, label noise is a natural outcome of dataset collection process and can occur in various domains, such as medical imaging [27, 28, 25], semantic segmentation [29, 30, 31], crowd-sourcing [32], social network tagging”
Structured Consistency Loss for semi-supervised semantic segmentation (2020, positive): “We adopt a state-of-the-art segmentation architecture, Deeplabv3+ with WideResNet38 from Zhu et al. [29].”
Self-Supervised CycleGAN for Object-Preserving Image-to-Image Domain Adaptation (2020, methodology): “Based on the widely-accepted protocol [37], we focus on 11 classes including bicyclist, building, car, pole, fence, pedestrian, road, sidewalk, sign, sky, and tree.”
KITTI-360: A Novel Dataset and Benchmarks for Urban Scene Understanding in 2D and 3D (2021, background): “[118] proposes a joint propagation strategy with synthesized training samples.”
Class-Incremental Learning for Semantic Segmentation Re-Using Neither Old Data Nor Old Labels (2020, background): “While current research is typically focused on performance improvements [5], [6], [7], [8] or knowledge transfer to new domains [9], [10], [11], the number of segmented classes is usually assumed to be constant.”
Scaling Wide Residual Networks for Panoptic Segmentation (2020, background): “Speciﬁcally, the Wide-ResNet-38 (WR-38) [92], reﬁned by several human-crafted networks with extensive experiments, has been the de facto network back-bone for semantic segmentation [63, 73, 105, 82, 52], and instance segmentation [55, 32] on Cityscapes leader-board [14].”
RobustNet: Improving Domain Generalization in Urban-Scene Segmentation via Instance Selective Whitening (2021, background): “Domain adaptation and generalization It is well known that signiﬁcant labeling efforts are required so as to ensure the reliable performance of various tasks such as semantic segmentation [35, 1, 4, 68, 8].”