Non-local Neural Networks
By X. Wang, Ross B. Girshick, Abhinav Kumar Gupta, Kaiming He. 2017
10,287 citations recorded by the source corpus.
Both convolutional and recurrent operations are building blocks that process one local neighborhood at a time. In this paper, we present non-local operations as a generic family of building blocks for capturing long-range dependencies. Inspired by the classical non-local means method [4] in computer vision, our non-local operation computes the response at a position as a weighted sum of the features at all positions. This building block can be plugged into many computer vision architectures. On the task of video classification, even without any bells and whistles, our nonlocal models can compete or outperform current competition winners on both Kinetics and Charades datasets. In static image recognition, our non-local models improve object detection/segmentation and pose estimation on the COCO suite of tasks. Code will be made available.
Subfields: Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Ming-Hsuan Yang
- Xiaogang Wang
- Trevor Darrell
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Model Architectures
- Quoc V. Le
- Alan L. Yuille
- Jianfeng Gao
- Jiashi Feng
- Shuicheng Yan
- Yoshua Bengio
- Kaiming He
- Christoph Feichtenhofer
Citations in the corpus
- An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Inspired by NLP successes, multiple works try combining CNN-like architectures with self-attention (Wang et al., 2018; Carion et al., 2020), some replacing the convolutions entirely (Ramachandran et al., 2019; Wang et al., 2020a).”
- End-to-End Object Detection with Transformers (2020, background): “ention mechanisms [2] are neural network layers that aggregate information from the entire input sequence. Transformers introduced self-attention layers, which, similarly to Non-Local Neural Networks [49], scan through each element of a sequence and update it by aggregating information from the whol”
- Denoising Diffusion Probabilistic Models (2020, methodology): “We use self-attention at the 16× 16 feature map resolution [63, 60].”
- Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “In [40], Wang et al.”
- Exploring Self-Attention for Image Recognition (2020, background): “Others applied self-attention in specific modules that were appended to convolutional structures [35, 45].”
- Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, methodology): “Attention has since been extended to vision, giving significant boosts to image classification [6], object detection [37], semantic segmentation [42], video classification [87], and adversarial defense [89].”
- On the Relationship between Self-Attention and Convolutional Layers (2019, methodology): “Self-attention was first added to CNN by either using channel-based attention (Hu et al., 2018) or non-local relationships across the image (Wang et al., 2018).”
- Stand-Alone Self-Attention in Vision Models (2019, neutral): “We achieve this in two steps:”
- Local Relation Networks for Image Recognition (2019, methodology): “There are also some non-convolution methods [28, 24] that compute the aggregation weights in a bottom-up fashion, with the weights determined by the composability of a pixel pair.”
- Attention Augmented Convolutional Networks (2019, methodology): “Additionally, multiple attention mechanisms have been proposed for visual tasks to address the weaknesses of convolutions [17, 16, 7, 46, 45, 53].”
- CCNet: Criss-Cross Attention for Semantic Segmentation (2018, methodology): “Specifically, current state-of-the-art semantic segmentation approaches based on [31] generates a dense attention map which has H × W weights (in green).”
- Group Normalization (2018, methodology): “We use the ResNet-50 I3D baseline as described in [62].”
- Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, background): “The main difference between the non-local and the proposed BERT attention is that non-local concept [34] is preferred to be utilized not at the end of the architecture, but some preferred locations inside the architecture.”
- Early Convolutions Help Transformers See Better (2021, background): “[46] show that (single-headed) self-attention is a form of non-local means [2] and that integrating it into a ResNet [19] improves several tasks.”
- An Empirical Study of Training Self-Supervised Vision Transformers (2021, background): “There have been continuous efforts on generalizing Transformers to computer vision [43, 5, 38, 48, 6, 15].”
- Training data-efficient image transformers & distillation through attention (2020, background): “Nevertheless hybrid architectures that combine convnets and transformers, including the self-attention mechanism, have recently exhibited competitive results in image classification [53], detection [6, 26], video processing [42, 50], unsupervised object discovery [33], and unified text-vision tasks ”
- Panoptic Feature Pyramid Networks (2019, methodology): “Moreover, in our baseline we deliberately avoid orthogonal architecture improvements like Non-local [53] or SE [27], which would likely yield further gains.”
- CNN-Generated Images Are Surprisingly Easy to Spot… for Now (2019, background): “BigGAN has a monolithic, classconditional structure, is trained on very large batch sizes, and uses self-attention layers [42, 39].”
- On the Anatomy of MCMC-based Maximum Likelihood Learning of Energy-Based Models (2019, methodology): “We implement the same training procedure for a vanilla ConvNet and a network with non-local layers [28].”