Disentangled Non-Local Neural Networks

By Minghao Yin, Zhuliang Yao, Yue Cao, Xiu Li, Zheng Zhang, Stephen Lin, Han Hu. 2020

378 citations recorded by the source corpus.

The non-local block is a popular module for strengthening the context modeling ability of a regular convolutional neural network. This paper first studies the non-local block in depth, where we find that its attention computation can be split into two terms, a whitened pairwise term accounting for the relationship between two pixels and a unary term representing the saliency of every pixel. We also observe that the two terms trained alone tend to model different visual clues, e.g. the whitened pairwise term learns within-region relationships while the unary term learns salient boundaries. However, the two terms are tightly coupled in the non-local block, which hinders the learning of each. Based on these findings, we present the disentangled non-local block, where the two terms are decoupled to facilitate learning for both terms. We demonstrate the effectiveness of the decoupled design on various tasks, such as semantic segmentation on Cityscapes, ADE20K and PASCAL Context, object detection on COCO, and action recognition on Kinetics.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

In Model Architectures

Citations in the corpus

Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, background): “The selfattention layers can complement backbones [61, 7, 3, 65, 21, 68, 51] or head networks [29, 24] by providing the capability to encode distant dependencies or heterogeneous interactions.”
K-Net: Towards Unified Image Segmentation (2021, background): “Based on this framework, many studies focus on enhancing the feature representation through dilated convolution [7–9], pyramid pooling [57, 67], context representations [64, 65], and attention mechanisms [32, 63, 68].”
Segmenter: Transformer for Semantic Segmentation (2021, methodology): “State-of-the-art methods deploy Fully Convolutional Networks (FCN) [42] and achieve impressive results on challenging segmentation benchmarks [9, 21, 53, 54, 56, 58, 61].”
Designing Network Design Strategies Through Gradient Path Analysis (2022, background): “For example, the pairwise relationship of non-local networks is easy to degenerate into unary information [37]; and (2) various specially designed arithmetic units are easy to cause difﬁculties in performance optimization.”
Attention mechanisms in computer vision: A survey (2021, background): “[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90”
A Simple Baseline for Open-Vocabulary Semantic Segmentation with Pre-trained Vision-Language Model (2021, methodology): “Fully convolutional network [36] and its variants [7,49,3], as a practical and straightforward approach to model the semantic segmentation as a pixel-wise classification problem, have dominated this field in the past few years.”
Exploring Cross-Image Pixel Contrast for Semantic Segmentation (2021, neutral): “We follow conventions [65, 81, 14, 76] for training hyper-parameters.”
You Only Learn One Representation: Unified Network for Multiple Tasks (2021, background): “Non-local networks [21, 4, 24] is another way to obtain attention, and it mainly extracts pair-wise attention in time and space.”
CMX: Cross-Modal Fusion for RGB-X Semantic Segmentation With Transformers (2022, methodology): “Additional FCNbased methods improve dense semantic segmentation by refining context priors [45], [46], [47], [48], [49], [50], [51], enhancing boundary cues [52], [53], [54], and appending various attention modules [55], [56], [57], [58], [59] to boost the performance.”
Video Swin Transformer (2021, background): “DNL [42] on the contrary attempts to alleviate this degeneration problem by a disentangled design that allows learning of different contexts for different pixels while preserving the shared global context.”
A Review on Deep Learning in UAV Remote Sensing (2021, neutral): “Still, as of today, the CGnet [35] and DLNet [47] are considered the state-of-art methods for semantic segmentation.”