CCNet: Criss-Cross Attention for Semantic Segmentation

By Zilong Huang, Xinggang Wang, Lichao Huang, Chang Huang, Yunchao Wei, Humphrey Shi. 2018

3,049 citations recorded by the source corpus.

Full-image dependencies provide useful contextual information to benefit visual understanding problems. In this work, we propose a Criss-Cross Network (CCNet) for obtaining such contextual information in a more effective and efficient way. Concretely, for each pixel, a novel criss-cross attention module in CCNet harvests the contextual information of all the pixels on its criss-cross path. By taking a further recurrent operation, each pixel can finally capture the full-image dependencies from all pixels. Overall, CCNet is with the following merits: 1) GPU memory friendly. Compared with the non-local block, the proposed recurrent criss-cross attention module requires 11x less GPU memory usage. 2) High computational efficiency. The recurrent criss-cross attention significantly reduces FLOPs by about 85% of the non-local block in computing full-image dependencies. 3) The state-of-the-art performance. We conduct extensive experiments on popular semantic segmentation benchmarks including Cityscapes, ADE20K, and instance segmentation benchmark COCO. In particular, our CCNet achieves the mIoU score of 81.4 and 45.22 on Cityscapes test set and ADE20K validation set, respectively, which are the new state-of-the-art results. The source code is available at https://github.com/speedinghzl/CCNet.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Efficient attention methods are proposed to reduce its complexity [76,42,56].”
Axial Attention in Multidimensional Transformers (2019, background): “Operations similar to unmasked axial attention have been proposed in other contexts in computer vision (Huang et al., 2019).”
VideoGPT: Video Generation using VQ-VAE and Transformers (2021, background): “, 2015) along with axial attention (Huang et al., 2019; Ho et al., 2019b) for the autoencoder in VQ-VAE, learning a downsampled set of discrete latents from raw pixels of the video frames.”
OneFormer: One Transformer to Rule Universal Image Segmentation (2022, background): “More recent works [26, 27, 44, 56] have shown the success of transformer-based methods in semantic segmentation following its success in language and vision [3, 49].”
Masked-attention Mask Transformer for Universal Image Segmentation (2021, methodology): “Follow-up methods find context to play an important role for precise per-pixel classification and focus on designing customized context modules [7,8,63] or self-attention variants [21,26,45,55,61,64].”
Per-Pixel Classification is Not All You Need for Semantic Segmentation (2021, methodology): “Modern semantic segmentation models focus on aggregating long-range context in the final feature map: ASPP [7, 8] uses atrous convolutions with different atrous rates; PPM [52] uses pooling operators with different kernel sizes; DANet [19], OCNet [51], and CCNet [23] use different variants of non-lo”
Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, background): “, Axial Attention[3], CCNet[4]) Efficient Sparse Attention in Image Domain”
Efficient Transformers: A Survey (2020, methodology): “Following which, we move on to models that utilize combinations of patterns such as Sparse Transformers (Child et al., 2019), CCNet (Huang et al., 2019), and Axial Transformers (Ho et al., 2019).”
MaX-DeepLab: End-to-End Panoptic Segmentation with Mask Transformers (2020, background): “Both classes of methods have boosted various vision tasks such as image classification [18, 5, 72, 37, 57, 89, 28], object detection [91, 80, 72, 36, 10, 108], semantic segmentation [15, 106, 39, 29, 109, 107], video recognition [91, 18], image generation [73, 35], and panoptic segmentation [89].”
Highly accurate protein structure prediction with AlphaFold (2021, positive): “In particular, we add an extra logit bias to axial attention 31 to include the ‘missing edge’ of the triangle and we define a non-attention update operation ‘triangle multiplicative update’ that uses two edges to update the missing third edge (see Supplementary Methods 1.6.5 for details).”
MSA Transformer (2021, methodology): “The architecture interleaves attention across the rows and columns of the alignment as in axial attention (Ho et al., 2019; Huang et al., 2019).”
Representation Compensation Networks for Continual Semantic Segmentation (2022, background): “Benefiting from its strong ability to aggregate features and model long-range dependency, strip pooling shines in many fully-supervised semantic segmentation models [38, 42].”
PLOP: Learning without Forgetting for Continual Semantic Segmentation (2020, background): “Examples of recent works exploiting contextual information include attention mechanisms [69, 76, 22, 30, 68, 61, 73], and fixed-scale aggregation [75, 12, 11, 72].”
Deep High-Resolution Representation Learning for Visual Recognition (2019, neutral): “0:8 78:5 CFNet [173] D-ResNet-101 79:6 - - - Auto-DeepLab [95] - 80:4 - - - DenseASPP [181] WDenseNet-161 80:6 59:1 90:9 78:1 SVCNet [33] ResNet-101 81:0 - - - ANN [195] D-ResNet-101 81:3 - - - CCNet [61] D-ResNet-101 81:4 - - - DANet [43] D-ResNet-101 81:5 - - - HRNetV2 HRNetV2-W48 81:6 61:8 92:1 8”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “6 CCNet [26] ResNet-101 8× R 81.”