Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation

By Huiyu Wang, Yukun Zhu, Bradley Green, Hartwig Adam, Alan L. Yuille, Liang-Chieh Chen. 2020

826 citations recorded by the source corpus.

Convolution exploits locality for efficiency at a cost of missing long range context. Self-attention has been adopted to augment CNNs with non-local interactions. Recent works prove it possible to stack self-attention layers to obtain a fully attentional network by restricting the attention to a local region. In this paper, we attempt to remove this constraint by factorizing 2D self-attention into two 1D self-attentions. This reduces computation complexity and allows performing attention within a larger or even global region. In companion, we also propose a position-sensitive self-attention design. Combining both yields our position-sensitive axial-attention layer, a novel building block that one could stack to form axial-attention models for image classification and dense prediction. We demonstrate the effectiveness of our model on four large-scale datasets. In particular, our model outperforms all existing stand-alone self-attention models on ImageNet. Our Axial-DeepLab improves 2.8% PQ over bottom-up state-of-the-art on COCO test-dev. This previous state-of-the-art is attained by our small variant that is 3.8x parameter-efficient and 27x computation-efficient. Axial-DeepLab also achieves state-of-the-art results on Mapillary Vistas and Cityscapes.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, methodology): “Inspired by NLP successes, multiple works try combining CNN-like architectures with self-attention (Wang et al., 2018; Carion et al., 2020), some replacing the convolutions entirely (Ramachandran et al., 2019; Wang et al., 2020a).”
VOLO: Vision Outlooker for Visual Recognition (2021, background): “Second, outlook attention adopts a sliding window mechanism to locally encode token representations at fine level, and to some extent preserves the crucial positional information for vision tasks [25, 55].”
Perceiver: General Perception with Iterative Attention (2021, methodology): “…et al., 2018; Ramachandran et al., 2019; Zhao et al., 2020; Sukhbaatar et al., 2019), using non-local, non-dense attention patterns (Ho et al., 2019; Wang et al., 2020a; Belt-agy et al., 2020; Child et al., 2019; Correia et al., 2019; Ye et al., 2019; Roy et al., 2020), approximating or otherwise…”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, methodology): “The state-of-theart method in this regime, Axial-DeepLab [89], along with other box-free methods [100, 21, 11], predicts pixel-wise offsets to pre-defined instance centers.”
Highly accurate protein structure prediction with AlphaFold (2021, background): “In parallel, the success of attention-based networks for language processing 52 and, more recently, computer vision 31,53 has inspired the exploration of attention-based methods for interpreting protein sequences 54–56 .”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Note that while the first scale of Local POD (1/2) is equivalent to POD and models long-range dependencies, which are important for segmentation [64, 31, 53, 28], the subsequent scales (s = 1/2(1), 1/2(2) .”
Segmenter: Transformer for Semantic Segmentation (2021, methodology): “The success of transformers in NLP has inspired several methods in computer vision combining CNNs with forms of selfattention to address object detection [6], semantic segmentation [49], panoptic segmentation [48], video processing [50] and few-shot classification [16].”
Is Space-Time Attention All You Need for Video Understanding? (2021, background): “A decomposed attention over the two spatial axes of the image was proposed in (Ho et al., 2019; Huang et al., 2019; Wang et al., 2020b) and our (T+W+H) adds a third dimension (time) for the case of video.”
LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “They are recently enjoying a resurgence of popularity with many works modifying the popular Transformer architecture for sequential processing applications (Katharopoulos et al., 2020; Wang et al., 2020b; Choromanski et al., 2020; Xiong et al., 2021).”
Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, methodology): “Other forms of non-global pixel interaction can also be implemented efficiently [6, 18, 54, 1].”
DeepViT: Towards Deeper Vision Transformer (2021, methodology): “Transformers [38] are initially used for machine translation which replace the recurrence and convolutions entirely with self-attention mechanisms [28, 14, 48, 11, 39, 17] and achieve outstanding performance.”
Vision Permutator: A Permutable MLP-Like Architecture for Visual Recognition (2021, methodology): “[23], [31] leverages axial attention to process representations by factorizing multidimensional”
Shuffle Transformer: Rethinking Spatial Shuffle for Vision Transformer (2021, methodology): “ods [21, 18, 31, 38, 7] and non-overlapping window-based methods [28, 19].”