Local Relation Networks for Image Recognition

By Han Hu, Zheng Zhang, Zhenda Xie, Stephen Lin. 2019

564 citations recorded by the source corpus.

The convolution layer has been the dominant feature extractor in computer vision for years. However, the spatial aggregation in convolution is basically a pattern matching process that applies fixed filters which are inefficient at modeling visual elements with varying spatial distributions. This paper presents a new image feature extractor, called the local relation layer, that adaptively determines aggregation weights based on the compositional relationship of local pixel pairs. With this relational approach, it can composite visual elements into higher-level entities in a more efficient manner that benefits semantic inference. A network built with local relation layers, called the Local Relation Network (LR-Net), is found to provide greater modeling capacity than its counterpart built with regular convolution on large-scale recognition tasks such as ImageNet classification.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Quoc V. Le
3. Alan L. Yuille
4. Dacheng Tao
5. Jiashi Feng
6. Luc Van Gool
7. Trevor Darrell
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Luc Van Gool
2. Dacheng Tao
3. Ming-Hsuan Yang
4. Xiaogang Wang
5. Trevor Darrell
6. Alan L. Yuille
7. Yi Yang
8. Chunhua Shen

In Model Architectures

1. Jianfeng Gao
2. Alan L. Yuille
3. Lu Yuan
4. Jianwei Yang
5. Jiashi Feng
6. Quoc V. Le
7. Liang-Chieh Chen
8. Chunhua Shen

Citations in the corpus

• An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Such local multi-head dot-product self attention blocks can completely replace convolutions (Hu et al., 2019; Ramachandran et al., 2019; Zhao et al., 2020).”
• Visual Transformers: Token-based Image Representation and Processing for Computer Vision (2020, background): “[18, 13, 21, 17, 15] on the other hand restrict receptive field of self-attention and use it in a convolutional manner.”
• Exploring Self-Attention for Image Recognition (2020, background): “[13] explored alternatives to the dot product, but these alternatives operated on scalar weights that were likewise shared across channels.”
• Axial-DeepLab: Stand-Alone Axial-Attention for Panoptic Segmentation (2020, background): “Recently, stacking attention layers as stand-alone models without any spatial convolution has been proposed [68,38] and shown promising results.”
• Stand-Alone Self-Attention in Vision Models (2019, neutral): “We achieve this in two steps:”
• Swin Transformer: Hierarchical Vision Transformer using Shifted Windows (2021, methodology): “Our Swin-T architecture brings consistent +3.4∼4.2 box AP gains over ResNet-50, with slightly larger model size, FLOPs and latency.”
• Deformable DETR: Deformable Transformers for End-to-End Object Detection (2020, background): “Most works (Liu et al., 2018a; Parmar et al., 2018; Child et al., 2019; Huang et al., 2019; Ho et al., 2019; Hu et al., 2019; Parmar et al., 2019; Qiu et al., 2019; Beltagy et al., 2020; Ainslie et al., 2020; Zaheer et al., 2020) follow this paradigm.”
• Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformers (2020, background): “LRNet [23] and standalone networks [40] explore local self-attention to avoid the heavy computation brought by global self-attention.”
• CvT: Introducing Convolutions to Vision Transformers (2021, background): “The local relation networks [17] adapts its weight aggregation based on the compositional relations (similarity) between pixels/features within a local window, in contrast to convolution layers which employ fixed aggregation weights over spatially neighboring input feature.”
• VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Recent studies showed that convolution-free, specially designed all-attention models can match CNNs’ performance on image recognition tasks [112, 48, 79].”
• LambdaNetworks: Modeling Long-Range Interactions Without Attention (2021, background): “5) Local relative self-attention (Hu et al., 2019) 23.”
• Bottleneck Transformers for Visual Recognition (2021, methodology): “Being a hybrid model, BoTNet differs from pure attention models such as SASA [49], LRNet [33], SANet [68], Axial-SASA [31, 62] and ViT [15].”
• Scaling Local Self-Attention for Parameter Efficient Visual Backbones (2021, methodology): “Self-attention has been viewed as a method to directly capture relationships between distant pixels [39, 19, 54].”
• CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, background): “There has also been a lot of interest in combining CNNs with different forms of self-attention [2, 32, 48, 31, 3, 17, 39].”
• DeepViT: Towards Deeper Vision Transformer (2021, methodology): “Transformers [38] are initially used for machine translation which replace the recurrence and convolutions entirely with self-attention mechanisms [28, 14, 48, 11, 39, 17] and achieve outstanding performance.”