Compact Generalized Non-local Network

By Kaiyu Yue, Ming Sun, Yuchen Yuan, Feng Zhou, Errui Ding, Fuxin Xu. 2018

176 citations recorded by the source corpus.

The non-local module is designed for capturing long-range spatio-temporal dependencies in images and videos. Although having shown excellent performance, it lacks the mechanism to model the interactions between positions across channels, which are of vital importance in recognizing fine-grained objects and actions. To address this limitation, we generalize the non-local module and take the correlations between the positions of any two channels into account. This extension utilizes the compact representation for multiple kernel functions with Taylor expansion that makes the generalized non-local module in a fast and low-complexity computation flow. Moreover, we implement our generalized non-local method within channel groups to ease the optimization. Experimental results illustrate the clear-cut improvements and practical applicability of the generalized non-local module on both fine-grained object recognition and video classification. Code is available at: this https URL.

Subfields: Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Dynamic Neural Networks: A Survey (2021, background): “Note that we leave out the detailed discussion on the self attention mechanism, which is widely studied in both NLP [6], [7] and CV fields [136], [137], [138] to re-weight features based on the similarity between queries and keys at different locations (temporal or spatial).”
Object-Contextual Representations for Semantic Segmentation (2019, methodology): “Double Attention and its related work [8,74,9,40,38,73,34,24] and ACFNet [74] group the pixels into a set of regions, and then augment the pixel representations by aggregating the region representations with the consideration of their context relations predicted by using the pixel representation.”
Efficient Attention: Attention with Linear Complexities (2018, methodology): “The table does not report results for CGNL with ResNet-101 and ResNeXt-101 since [27] did not report such results.”
Hierarchical Multi-Scale Attention for Semantic Segmentation (2020, background): “OCRNet [9], DANET [10], CFNet [11], OCNet [12] and other related work [13, 14, 15, 16, 17, 18, 19, 20] use such relationships to build better context.”
OCNet: Object Context Network for Scene Parsing (2018, methodology): “Some recent efforts (Huang et al., 2019; Yue et al., 2018; Zhu et al., 2019) propose different mechanisms to decrease the computation complexity and memory consumption of selfattention scheme.”
Involution: Inverting the Inherence of Convolution for Visual Recognition (2021, background): “Some works sparingly insert self-attention as plugin modules into the backbone neural network [4, 49] or attach them on the top of the backbone to extract high-level semantic relationships [3, 36], retaining the substratum of convolutional features.”
A Survey on Vision Transformer (2020, methodology): “For example, CGNL [140] applies the Taylor series of the RBF kernel function to approximate the pixel similarities.”
Multi-Agent Reinforcement Learning Based Frame Sampling for Effective Untrimmed Video Recognition (2019, background): “nonlocal network [47], compact generalized nonlocal network [54] and Nonlocal + GCN [48] bring great performance gain by leveraging extra network modules to capture feature-point-wise or ROI-wise spatial-temporal relations.”
Neural Architecture Search for Lightweight Non-Local Networks (2020, methodology): “[44] use Taylor expansion to optimize the non-local module; Cao et al .”
Attention mechanisms in computer vision: A survey (2021, unknown): “[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90”
Global Self-Attention Networks for Image Recognition (2020, background): “Different from the above works, Chen et al. (2018); Huang et al. (2019); Shen et al. (2018); Yue et al. (2018) made the global attention mechanism efficient by either removing the softmax normalization on the product of queries and keys and changing the order of matrix multiplications involved in…”
Not only Look, but also Listen: Learning Multimodal Violence Detection under Weak Supervision (2020, background): “Recently, self-attention networks [40,47,5,18] have been successfully applied in vision problems.”
OCNet: Object Context for Semantic Segmentation (2021, methodology): “For example, CGNL (Yue et al. 2018) (Compact Generalized Non-local) applies the Taylor series of the RBF kernel function to approximate the pair-wise similarities, RCCA (Huang et al. 2019) (Recurrent Criss-Cross Attention) applies two consecutive criss-cross attention to approximate the original sel”
LatentGNN: Learning Efficient Non-local Relations for Visual Recognition (2019, background): “Unlike previous non-local relation networks (Wang et al., 2018; Yue et al., 2018), our strategy proposes a novel graph structure for fast message updates and a ﬂexible graph afﬁnity representation for learning the non-local relations in a task-driven manner.”