Attentional Pooling for Action Recognition

By Rohit Girdhar, Deva Ramanan. Neural Information Processing Systems, 2017

338 citations recorded by the source corpus.

We introduce a simple yet surprisingly powerful model to incorporate attention in action recognition and human object interaction tasks. Our proposed attention module can be trained with or without extra supervision, and gives a sizable boost in accuracy while keeping the network size and computational cost nearly the same. It leads to significant improvements over state of the art base architecture on three standard action recognition benchmarks across still images and videos, and establishes new state of the art on MPII dataset with 12.5% relative improvement. We also perform an extensive analysis of our attention module both empirically and analytically. In terms of the latter, we introduce a novel derivation of bottom-up and top-down attention as low-rank approximations of bilinear pooling methods (typically used for fine-grained classification). From this perspective, our attention formulation suggests a novel characterization of action recognition as a fine-grained recognition problem.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/90bb40d96fcd16129eb85ce5dc4ee3b2380d74c8

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Robust Attentional Aggregation of Deep Feature Sets for Multi-view 3D Reconstruction (2018, methodology): “Comparing with the widely used feed-forward attention mechanisms for visual recognition (Hu et al. 2018; Rodríguez et al. 2018; Liu et al. 2018; Sarafianos et al. 2018; Girdhar and Ramanan 2017), our FASet algorithm is the first to investigate and improve the robustness of attention modules to dynam”
Rethinking Spatiotemporal Feature Learning For Video Understanding (2017, methodology): “Since then, many works on video understanding follow the same multi-stream 2D CNN design, and have made improvements in terms of backbone architecture [10, 52, 30, 12], fusion of the streams [11, 8, 9, 58] and exploiting richer temporal structures [6, 54, 53].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, background): “Early work studied hybrid models consisting of both convolutions and attention modules [94, 100, 40, 111].”
ViViT: A Video Vision Transformer (2021, methodology): “It is thus analogous to CNN architectures such as [24, 34, 72, 86] which ﬁrst extract per-frame features, and then aggregate them into a ﬁnal representation before classifying them.”
Attention mechanisms in computer vision: A survey (2021, neutral): “[8], [9], [15], [21], [31], [32], [34], [35] , [22], [26], [62], [63], [64], [65], [66], [67] , [41], [68], [69], [70], [71], [72], [73], [74] , [8], [34], [42], [43], [75], [76], [77], [78] , [27], [44], [45], [46], [79], [80], [81], [82] , [61], [83], [84], [85], [86], [87], [88], [89] , [47], [90”
Attention, please! A survey of neural attention models in deep learning (2021, neutral): “(2019), Girdhar and Ramanan (2017), Song et al.”
Detecting Hands and Recognizing Physical Contact in the Wild (2020, methodology): “The spatial attention method shares similarities with several visual attention methods that have gained much interest over the past years [1, 5, 9, 14, 20, 21, 24, 31, 32, 36].”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “spatial attention [78], [79], [80], [81], temporal attention [82], [83], and both spatial and temporal attention [84], [85], have”
HAKE: Human Activity Knowledge Engine (2019, neutral): “Method mAP AlexNet+SVM [13] 19.4 R*CNN [7] 28.5 Girdhar & Ramanan [6] 34.6 Mallya & Lazebnik [19] 36.1 Pairwise [14] 39.9 Pairwise [14]+ Table 3.”
Adversarial Cross-Domain Action Recognition with Co-Attention (2019, methodology): “…ﬁrst select segments that are critical for cross-domain action recognition by investigating temporal attention, which is a widely used technique in action recognition (Sharma, Kiros, and Salakhutdinov 2015; Girdhar and Ramanan 2017) that helps the model focus on segments more related to the action.”
Anticipative Video Transformer (2021, background): “The general idea of self-attention in vision dates back to non-local means [9], and is incorporated into contemporary network architectures as non-local blocks [10, 54, 90, 92] and gating mechanisms [30, 45, 60, 94].”
iCAN: Instance-Centric Attention Network for Human-Object Interaction Detection (2018, methodology): “To address this limitation, recent works leverage end-to-end trainable attention modules for action recognition [9] or image classification [20].”
Compressed Video Action Recognition (2017, neutral): “6 Attentional Pooling [10] 52.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “Since then, many video classiﬁcation methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
Video Action Transformer Network (2018, background): “Attention for videos has been pursued in various forms, including gating or second order pooling [10, 27, 28, 45] guided by human pose or other primitives [4, 5, 10, 11], recurrent models [34] and self-attention [44].”
A2-Nets: Double Attention Networks (2018, background): “We note that similar understandings were presented in [7], in which they proposed a rank-1 approximation of a bilinear pooling operation associated with a fully connected classifier.”
Timeception for Complex Action Recognition (2018, methodology): “A widely used approach is statistical pooling: max and average pooling [18, 19], attention pooling [6], rank pooling [20], dynamic images [21] and context gating [7], to name a few.”
FSA-Net: Learning Fine-Grained Structure Aggregation for Head Pose Estimation From a Single Image (2019, methodology): “Compared with other pooling methods using attention such as CBAM [41] and Attentional Pooling [15], our method has the following differences with them.”
PoTion: Pose MoTion Representation for Action Recognition (2018, positive): “Overall, we outperform all existing approaches on all datasets, including methods that leverage pose [13, 48] or capture long-term motion [36].”