Hierarchical Multi-Scale Attention for Semantic Segmentation

By Andrew Tao, Karan Sapra, Bryan Catanzaro. 2020

500 citations recorded by the source corpus.

Multi-scale inference is commonly used to improve the results of semantic segmentation. Multiple images scales are passed through a network and then the results are combined with averaging or max pooling. In this work, we present an attention-based approach to combining multi-scale predictions. We show that predictions at certain scales are better at resolving particular failures modes, and that the network learns to favor those scales for such cases in order to generate better predictions. Our attention mechanism is hierarchical, which enables it to be roughly 4x more memory efficient to train than other recent approaches. In addition to enabling faster training, this allows us to train with larger crop sizes which leads to greater model accuracy. We demonstrate the result of our method on two datasets: Cityscapes and Mapillary Vistas. For Cityscapes, which has a large number of weakly labelled images, we also leverage auto-labelling to improve generalization. Using our approach we achieve a new state-of-the-art results in both Mapillary (61.1 IOU val) and Cityscapes (85.1 IOU test).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

EDTER: Edge Detection with Transformer (2022, background): “More recently, some approaches improve edge detection [22, 36, 37, 65, 66], segmentation [8, 54, 70], and object detection [35] by using hierarchical multi-scale features.”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “In recent years, the introduction of Convolutional Neural Networks (CNNs) has addressed semantic segmentation in a traditional framework, where all classes are known beforehand and learned at once [61, 73, 11].”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “Network (UperNet) [136], segmentation by re-training and self-training [137], densely connected neural architecture search [138], hierarchical multiscale attention [139], Efficient RGB-D Semantic Segmentation (ESA-Net) [140], Iterative Pyramid Contexts [141], and Learning Dynamic Routing for Semanti”
HRDA: Context-Aware High-Resolution Domain-Adaptive Semantic Segmentation (2022, unknown): “For that purpose, we newly extend scale attention [5, 56] to UDA and reveal its significance for UDA by improving the adaptation process across different object scales.”
Vision Transformer Adapter for Dense Predictions (2022, methodology): “As shown in Table 11, our ViT-Adapter-L achieves 85.2 multi-scale mIoU on the test set, which is slightly better than HRNetV2+OCR+HMS [70] that uses extra coarsely annotated data.”
AdaShare: Learning What To Share For Efficient Deep Multi-Task Learning (2019, methodology): “Following [47], we sampled 5 representative tasks out of 26 tasks for our experiments, namely Semantic Segmentation, Surface Normal Prediction, Depth Prediction, Keypoint Detection and Edge Detection.”
Tackling Catastrophic Forgetting and Background Shift in Continual Semantic Segmentation (2021, background): “Examples of recent works exploiting contextual information include attention mechanisms [5], [6], [33], [34], [35], [36], [37], and fixed-scale aggregation [7], [28], [38], [39].”
Adaptive Prototype Learning and Allocation for Few-Shot Segmentation (2021, background): “Recent breakthroughs in semantic segmentation have mainly come from multi-scale feature aggregation [43, 2, 35, 10] or attention mechanisms [7, 38, 15, 28, 3, 44, 41].”
InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions (2022, methodology): “Following common practices [69,83,108], we first pre-train on Mapillary Vistas [109] and then fine-tune on Cityscapes for 80k iterations, respectively.”
Attentional Feature Fusion (2020, background): “In the ﬁrst type, the features at multiple scales or their concatenated result are fed into the attention module to generate multi-scale attention maps, while the scale of feature context aggregation inside the attention module remains single [2, 3, 46, 6, 36, 41].”
DAFormer: Improving Network Architectures and Training Strategies for Domain-Adaptive Semantic Segmentation (2021, methodology): “For the DAFormer architecture, we use the MiT-B5 encoder [83], which produces a feature pyramid with C = [64, 128, 320, 512].”
Polarized Self-Attention: Towards High-quality Pixel-wise Regression (2021, methodology): “To our knowledge, the current state-of-the-art results by single models were produced by HRNet-OCR(MA) [41] on the Cityscapes validation set [9](19 classes, input image size 1024 × 2048, output mask size 1024 × 2048).”
HyperSeg: Patch-wise Hypernetwork for Real-time Semantic Segmentation (2020, methodology): “Recent work added capacity [5, 6] and attention mechanisms [20, 45, 49] to improve performance.”