Full-Resolution Residual Networks for Semantic Segmentation in Street Scenes

By Tobias Pohlen, Alexander Hermans, Markus Mathias, Bastian Leibe. 2016

589 citations recorded by the source corpus.

Semantic image segmentation is an essential component of modern autonomous driving systems, as an accurate understanding of the surrounding scene is crucial to navigation and action planning. Current state-of-the-art approaches in semantic image segmentation rely on pre-trained networks that were initially developed for classifying images as a whole. While these networks exhibit outstanding recognition performance (i.e., what is visible?), they lack localization accuracy (i.e., where precisely is something located?). Therefore, additional processing steps have to be performed in order to obtain pixel-accurate segmentation masks at the full image resolution. To alleviate this problem we propose a novel ResNet-like architecture that exhibits strong localization and recognition performance. We combine multi-scale context with pixel-level accuracy by using two processing streams within our network: One stream carries information at the full image resolution, enabling precise adherence to segment boundaries. The other stream undergoes a sequence of pooling operations to obtain robust features for recognition. The two streams are coupled at the full image resolution using residuals. Without additional processing steps and without pre-training, our approach achieves an intersection-over-union score of 71.8% on the Cityscapes dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Rethinking Atrous Convolution for Semantic Image Segmentation (2017, methodology): “Second, the encoder-decoder structure [3, 63, 21, 47, 62, 60, 35] exploits multi-scale features from the encoder part and re-covers the spatial resolution from the decoder part.”
Deep High-Resolution Representation Learning for Visual Recognition (2019, background): “An extension of U-Net, full-resolution residual network [114], introduces an extra full-resolution stream that carries information at the full image resolution, to replace the skip connections, and each unit in the downsample and upsample subnetworks receives information from and sends information…”
Segmenter: Transformer for Semantic Segmentation (2021, background): “Consecutive approaches [1, 4, 34, 41, 43] upsample high-level feature maps and combine them with low-level feature maps during decoding to both capture global information and recover sharp object boundaries.”
Panoptic-DeepLab: A Simple, Strong, and Fast Baseline for Bottom-Up Panoptic Segmentation (2019, background): “The loss improves over bootstrapped cross entropy loss [72, 7, 59] by weighting each pixel differently.”
Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerialization (2019, background): “Authors of state-of-the-art models cite memory as a limiting factor in image classification [20, 25, 34], semantic segmentation [8, 45], and NLP [11, 13, 36].”
ExFuse: Enhancing Feature Fusion for Semantic Segmentation (2018, neutral): “A common way of feature fusion [27,2,28,6,7,8,11] is to formulate as a residual form:”
Weakly- and Semi-Supervised Panoptic Segmentation (2018, background): “Moreover, [46] and [47] showed improved results by respectively sampling only 6% and 12% of the hardest pixels, instead of all of them, in fully-supervised training.”
Multi-scale Context Intertwining for Semantic Segmentation (2018, methodology): “performance, some networks [35, 17, 19, 21] utilize an Encoder-Decoder (ED) network to gradually combine adjacent feature maps along the top-down hierarchy of a common FCN architecture, propagating the semantic information from the low-resolution feature maps to the high-resolution feature maps and ”
ERFNet: Efficient Residual Factorized ConvNet for Real-Time Semantic Segmentation (2018, methodology): “However, these approaches achieve top results by highly increasing the complexity and resources of their networks: RefineNet [9] and FRRN [10] employ large ResNets-like architectures in multiple pipelines that work with high resolution feature maps; Deeplabv2 [8] uses a large ResNet (101-layers) to ”
Computer Vision for Autonomous Vehicles: Problems, Datasets and State-of-the-Art (2017, background): “While Pohlen et al. (2016) achieve competitive results on Cityscapes (Table 4a),”
DFANet: Deep Feature Aggregation for Real-Time Semantic Segmentation (2019, unknown): “Results of existing real-time methods, including ICNet[33], ENet[22], SQ[25], SegNet[1], FRRN[24], FCN-8S[19], TwoColumn[27], BiSeNet[29].”
Encoder–decoder network for pixel‐level road crack detection in black‐box images (2019, methodology): “…eﬃcient and safe for monitoring the road pavement conditions compared with manual inspections or those involving specialized vehicles (Haas, 1996; Radopoulou & Brilakis, 2016; Sun, Kamaliardakani, & Zhang, 2015; Zalama, Gómez-García-Bermejo, Medina, & Llamas, 2014; Zou, Cao, Li, Mao, & Wang,…”
Segmentation-Aware Convolutional Networks Using Local Attention Masks (2017, methodology): “The low-resolution issue has received substantial attention: for instance methods have been proposed for replacing the subsampling layers with resolution-preserving alternatives such as atrous convolution [9, 58, 43], or restoring the lost resolution via upsampling stages [39, 34].”
Survey on semantic segmentation using deep learning techniques (2019, background): “Pohlen et al. [44] proposed a full-resolution residual network (FRRN) with strong localization and recognition performance for semantic segmentation.”