RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation

By Guosheng Lin, Anton Milan, Chunhua Shen, Ian David Reid. 2016

3,126 citations recorded by the source corpus.

Recently, very deep convolutional neural networks (CNNs) have shown outstanding performance in object recognition and have also been the first choice for dense classification problems such as semantic segmentation. However, repeated subsampling operations like pooling or convolution striding in deep CNNs lead to a significant decrease in the initial image resolution. Here, we present RefineNet, a generic multi-path refinement network that explicitly exploits all the information available along the down-sampling process to enable high-resolution prediction using long-range residual connections. In this way, the deeper layers that capture high-level semantic features can be directly refined using fine-grained features from earlier convolutions. The individual components of RefineNet employ residual connections following the identity mapping mindset, which allows for effective end-to-end training. Further, we introduce chained residual pooling, which captures rich background context in an efficient manner. We carry out comprehensive experiments and set new state-of-the-art results on seven public datasets. In particular, we achieve an intersection-over-union score of 83.4 on the challenging PASCAL VOC 2012 dataset, which is the best reported result to date.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “EncNet [39] achieves previous best performance among the methods and utilizes global pooling with image-level super-vision to collect image-level context information.”
Unified Perceptual Parsing for Scene Understanding (2018, background): “More recently, RefineNet [15] uses a coarse-to-fine architecture which exploits all information available along the down-sampling process.”
DeepSetNet: Predicting Sets with Deep Neural Networks (2016, background): “Deep convolutional (CNN) and recurrent (RNN) neural networks now outperform traditional approaches in tasks like semantic segmentation [6, 25], image captioning [21] or object detection [27].”
Improved Techniques for Training Score-Based Generative Models (2020, methodology): “Here (ii) and (iii) do not affect the results much, but they are included because we hope to minimize the number of unnecessary changes to the standard RefineNet architecture (the original RefineNet blocks in [24] use max pooling and have no normalization layers).”
Diffusion Models Beat GANs on Image Synthesis (2021, background): “[20] found to substantially improve sample quality over the previous architectures [51, 27] used for denoising score matching.”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “Following up on the same dataset, Urooj and Borji [73] used the RefineNet-ResNet101 [38] to achieve the state-of-the-art hand segmentation performance at the time.”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, background): “In the recent years, we have witnessed substantial progress in this area [1], [2], [3], [4], [5], [6], mostly thanks to the introduction of fully convolutional networks [1] and to the integration of several other architectural components, such as attention models [7], [8], [9] or multi-scale fusion ”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Currently methods pay more attention to multi-scale feature aggregation [3, 34, 52,53,59,65,68,81].”
CARLA: An Open Urban Driving Simulator (2017, methodology): “The perception stack we describe here is built upon a semantic segmentation network based on RefineNet [17].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, background): “ReﬁneNet [90] improves the combination of upsam-pled representations and the representations of the same resolution copied from the downsample process.”
Segmenter: Transformer for Semantic Segmentation (2021, background): “Consecutive approaches [1, 3, 33, 39, 41] upsample high-level feature maps and combine them with low-level feature maps during decoding to both capture global information and recover sharp object boundaries.”
Context Encoding for Semantic Segmentation (2018, methodology): “We follow the prior work [5, 31, 39] to use the semantic labels of the most frequent 59 object categories plus background (60 classes in total).”
Deep Layer Aggregation (2017, background): “Prior work on ‘shallow-skip’ layers, e.g., U-Net (Ronneberger et al. 2015), FCN (Long et al. 2015) and FPN (Lin et al. 2017), provide connections that bypass traditional sequential layers.”