Attention to Scale: Scale-Aware Semantic Image Segmentation

By Liang-Chieh Chen, Yi Yang, Jiang Wang, Wei Xu, Alan L. Yuille. 2015

1,391 citations recorded by the source corpus.

Incorporating multi-scale features in fully convolutional neural networks (FCNs) has been a key element to achieving state-of-the-art performance on semantic image segmentation. One common way to extract multi-scale features is to feed multiple resized input images to a shared deep network and then merge the resulting features for pixelwise classification. In this work, we propose an attention mechanism that learns to softly weight the multi-scale features at each pixel location. We adapt a state-of-the-art semantic image segmentation model, which we jointly train with multi-scale input images and the attention model. The proposed attention model not only outperforms averageand max-pooling, but allows us to diagnostically visualize the importance of features at different positions and scales. Moreover, we show that adding extra supervision to the output at each scale is essential to achieving excellent performance when merging multi-scale features. We demonstrate the effectiveness of our model with extensive experiments on three challenging datasets, including PASCAL-Person-Part, PASCAL VOC 2012 and a subset of MS-COCO 2014.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “As a result, while graph convolutions require hundreds of nodes (128 nodes in [4], 340 in [25], 150 in [48]) to encode potential semantic concepts, our VT uses just 16 visual tokens and attains higher accuracy.”
CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “[7] made use of several attention masks to fuse feature maps or predictions from different branches.”
UPSNet: A Unified Panoptic Segmentation Network (2019, background): “Recent semantic segmentation methods that exploit deep convolutional feature extraction mainly approach this problem from either multi-scale feature aggregation [40, 26, 5, 14], or end-to-end structured prediction [2, 41, 1, 25, 4] perspectives.”
Fully Convolutional Instance-Aware Semantic Segmentation (2016, background): “The FCNs are extended with global context [28], multi-scale feature fusion [4], and deconvolution [31].”
Why Should We Add Early Exits to Neural Networks? (2020, methodology): “Apart from Inception, similar approaches have been successfully applied to, among others, face recognition [61], object detection [14], image segmentation [15], pose estimation [25], saliency detection [41], metric learning [71], visual attention prediction [69], and super resolution [34, 65].”
Multi-Scale Context Aggregation by Dilated Convolutions (2015, methodology): “‘DeepLab++’ stands for DeepLab-CRF-COCO-LargeFOV and ‘DeepLab-MSc++’ stands for DeepLab-MSc-CRF-LargeFOV-COCO-CrossJoint (Chen et al., 2015a).”
Pyramid Scene Parsing Network (2016, background): “2, FCN predicts the boat in the yellow box as a “car” based on its appearance.”
Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentation (2018, background): “, image pyramid) [18, 16, 58, 44, 11, 9] or those that adopt probabilistic graphical models (such as DenseCRF [37] with efficient inference algorithm [2]) [8, 4, 82, 44, 48, 55, 63, 34, 72, 6, 7, 9].”
Rethinking Atrous Convolution for Semantic Image Segmentation (2017, methodology): “First, the DCNN is applied to an image pyramid to extract features for each scale input [19, 17, 59, 46, 11, 10] where objects at different scales become prominent at different feature maps.”
UberNet: Training a Universal Convolutional Neural Network for Low-, Mid-, and High-Level Vision Using Diverse Datasets and Limited Memory (2016, methodology): “We therefore use a concatenation of the scores followed by a linear operation, as in the case of fusing the skip-layers described above, and leave the exploration of scale-aware processing [15] for the future.”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, background): “Both classes of methods have boosted various vision tasks such as image classification [18, 5, 72, 37, 57, 89, 28], object detection [91, 80, 72, 36, 10, 108], semantic segmentation [15, 106, 39, 29, 109, 107], video recognition [91, 18], image generation [73, 35], and panoptic segmentation [89].”
Residual Attention Network for Image Classification (2017, background): “Attention to scale [3] uses soft attention as a scale selection mechanism and gets state-of-the-art results in image segmentation task.”
Auto-DeepLab: Hierarchical Neural Architecture Search for Semantic Image Segmentation (2019, background): “It has been known that context information is crucial for pixel labeling tasks [26, 69, 37, 39, 16, 54, 14, 10].”
Uncertainty-aware Contrastive Distillation for Incremental Semantic Segmentation (2022, background): “In the recent years, we have witnessed substantial progress in this area [1], [2], [3], [4], [5], [6], mostly thanks to the introduction of fully convolutional networks [1] and to the integration of several other architectural components, such as attention models [7], [8], [9] or multi-scale fusion ”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some methods [14,22,32,37,38,50,55] is inspired by Non-local [85], utilizing attention mechanisms to establish connections between image contexts.”
Modeling the Background for Incremental Learning in Semantic Segmentation (2020, background): “by exploiting multiscale representations [19, 33], modeling spatial dependencies and contextual cues [6, 5, 8] or considering attention models [7].”
Searching for Efficient Multi-Scale Architectures for Dense Image Prediction (2018, background): “…(1) designing models that take as input an image pyramid so that large scale objects are captured by the downsampled image [26, 66, 23, 50, 13, 11], (2) designing models that contain encoder-decoder structures [2, 72, 49, 28, 64, 93, 96], or (3) designing models that employ a multi-scale…”
RefineNet: Multi-path Refinement Networks for High-Resolution Semantic Segmentation (2016, methodology): “We carry out experiments on the Person-Part dataset [8, 7] which provides pixel-level labels for six person parts including Head, Torso, Upper/Lower Arms and Upper/Lower Legs.”