Expectation-Maximization Attention Networks for Semantic Segmentation

By Xia Li, Zhisheng Zhong, Jianlong Wu, Yibo Yang, Zhouchen Lin, Hong Liu. 2019

645 citations recorded by the source corpus.

Self-attention mechanism has been widely used for various tasks. It is designed to compute the representation of each position by a weighted sum of the features at all positions. Thus, it can capture long-range relations for computer vision tasks. However, it is computationally consuming. Since the attention maps are computed w.r.t all other positions. In this paper, we formulate the attention mechanism into an expectation-maximization manner and iteratively estimate a much more compact set of bases upon which the attention maps are computed. By a weighted summation upon these bases, the resulting representation is low-rank and deprecates noisy information from the input. The proposed Expectation-Maximization Attention (EMA) module is robust to the variance of input and is also friendly in memory and computation. Moreover, we set up the bases maintenance and normalization methods to stabilize its training procedure. We conduct extensive experiments on popular semantic segmentation benchmarks including PASCAL VOC, PASCAL Context, and COCO Stuff, on which we set new records.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

K-Net: Towards Unified Image Segmentation (2021, background): “Based on this framework, many studies focus on enhancing the feature representation through dilated convolution [7–9], pyramid pooling [59, 69], context representations [66, 67], and attention mechanisms [32, 65, 70].”
Representation Compensation Networks for Continual Semantic Segmentation (2022, methodology): “Some methods [14,22,32,37,38,50,55] is inspired by Non-local [85], utilizing attention mechanisms to establish connections between image contexts.”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, background): “OCRNet [9], DANET [10], CFNet [11], OCNet [12] and other related work [13, 14, 15, 16, 17, 18, 19, 20] use such relationships to build better context.”
Strip Pooling: Rethinking Spatial Pooling for Scene Parsing (2020, background): “Besides, aggregating non-local context [36, 58, 29, 15, 7, 45, 21, 53, 66, 23, 14] is also effective for scene parsing.”
Image Segmentation Using Deep Learning: A Survey (2020, methodology): “Other applications of attention mechanisms to semantic segmentation include OCNet [92], which employs an object context pooling inspired by self-attention mechanism, ResNeSt: Split-Attention Networks [93], Height-driven Attention Networks [94], Expectation-Maximization Attention (EMANet) [95], Criss”
Scene Segmentation With Dual Relation-Aware Attention Network (2020, methodology): “Aˆ2Net [38] adopts weighted global pooling and weighted broadcasting to gather and distribute contextual information.”
SA-Net: Shuffle Attention for Deep Convolutional Neural Networks (2021, background): “…mechanisms most commonly used in computer vision: channel attention and spatial attention , both of which strengthen the original features by aggregating the same feature from all the positions with different aggregation strategies, transformations, and strengthening functions [5, 6, 7, 8, 9].”
OCNet: Object Context for Semantic Segmentation (2021, background): “There also exist some con-current efforts (Fu et al. 2019a; Zhang et al. 2019b; Huang et al. 2019; Li et al. 2019; Zhao et al. 2018) that exploit the semantic relations between pixels to construct the context, and our approach is different from most of them as we propose a simple yet effective…”
A Survey on Vision Transformer (2020, background): “PSANet [151], OCNet [139], DANet [38] and CFNet [147] are the first works that introduce the self-attention module to semantic segmentation task, which consider and augment the relation and similarity [146, 74, 46, 89, 130] between the contextual pixels.”
Visual Attention Methods in Deep Learning: An In-Depth Survey (2022, methodology): “In this regard, Li et al. [106] proposed using expectation maximization to build an attention method that iteratively forms a set of bases that compute the attention maps [29].”
Beyond Self-Attention: External Attention Using Two Linear Layers for Visual Tasks (2021, methodology): “[6] adopted expectationmaximization (EM) clustering to optimize self-attention.”
Attention mechanisms in computer vision: A survey (2021, methodology): “It was followed by a series of works such as EMANet [40], CCNet [41], HamNet [42] and the Stand-Alone Network [43], which improved speed, quality of results, and generalization capability.”
Multiattention Network for Semantic Segmentation of Fine-Resolution Remote Sensing Images (2020, background): “Therefore, substantial researches have been implemented which aim to alleviate the bottleneck to efficiency and push the boundaries of attention, including accelerating the generation process of the attention matrix [67, 72–74], pruning the structure of the attention block [75], and optimizing atten”
Rethinking Semantic Segmentation: A Prototype View (2022, background): “Since it was proposed, numerous efforts have been devoted to improving FCN, by, for example, enlarging the receptive field [15, 16, 25, 124, 128, 135]; strengthening context cues [4, 43, 47, 48, 56, 57, 70, 75, 77, 81, 90, 126, 128, 129, 132, 138, 141]; leveraging boundary information [6, 14, 27, 66”