Learning Spatio-Temporal Representation With Local and Global Diffusion

By Zhaofan Qiu, Ting Yao, Chong-Wah Ngo, Xinmei Tian, Tao Mei. 2019

185 citations recorded by the source corpus.

Convolutional Neural Networks (CNN) have been regarded as a powerful class of models for visual recognition problems. Nevertheless, the convolutional filters in these networks are local operations while ignoring the large-range dependency. Such drawback becomes even worse particularly for video recognition, since video is an information-intensive media with complex temporal variations. In this paper, we present a novel framework to boost the spatio-temporal representation learning by Local and Global Diffusion (LGD). Specifically, we construct a novel neural network architecture that learns the local and global representations in parallel. The architecture is composed of LGD blocks, where each block updates local and global features by modeling the diffusions between these two representations. Diffusions effectively interact two aspects of information, i.e., localized and holistic, for more powerful way of representation learning. Furthermore, a kernelized classifier is introduced to combine the representations from two aspects for video recognition. Our LGD networks achieve clear improvements on the large-scale Kinetics-400 and Kinetics-600 video classification datasets against the best competitors by 3.5% and 0.7%. We further examine the generalization of both the global and local representations produced by our pre-trained LGD networks on four different benchmarks for video action recognition and spatio-temporal action detection tasks. Superior performances over several state-of-the-art techniques on these benchmarks are reported.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Multiview Transformers for Video Recognition (2022, neutral): “76.3 – – – I3D NL [74] 77.7 93.3 10 × 3 10.77 VidTR-L [83] 79.1 93.9 10 × 3 10.53 LGD-3D R101 [52] 79.4 94.4 – – SlowFast R101-NL [23] 79.8 93.9 10 × 3 7.02 X3D-XXL [22] 80.4 94.6 SlowFast R50 [23,77] 61.7 – TimeSformer-HR [6] 62.5 – VidTR [83] 63.0 – ViViT-L FE [3] 65.9 89.9 MViT [21] 67.7 90.9…”
Is Space-Time Attention All You Need for Video Understanding? (2021, neutral): “…N/A I3D+NL (Wang et al., 2018b) 77.7 93.3 10.8 ip-CSN-152 (Tran et al., 2019) 77.8 92.8 3.2 CorrNet (Wang et al., 2020a) 79.2 N/A 6.7 LGD-3D-101 (Qiu et al., 2019) 79.4 94.4 N/A SlowFast (Feichtenhofer et al., 2019b) 79.8 93.9 7.0 X3D-XXL (Feichtenhofer, 2020) Is Space-Time Attention All You…”
ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learning for Action Recognition (2022, background): “Model Pretrain #Frames GFlops Top-1 Top-5 Methods with full-finetuning LGD[53] IN-1K 128×N/A N/A 79.”
TEA: Temporal Excitation and Aggregation for Action Recognition (2020, methodology): “In both tables, the first compartment contains the methods based on 3D CNNs or the mixup of 2D and 3D CNNs, and the methods in the second compartment are all based on 2D or (2+1)D CNNs. Due to the high computation costs of 3D CNNs, the FLOPs of methods in the first compartment are typically higher t”
TDN: Temporal Difference Networks for Efficient Action Recognition (2020, background): “Deep neural networks have witnessed great progress for action recognition in videos [14, 29, 38, 31, 6, 26, 37].”
Attention Bottlenecks for Multimodal Fusion (2021, background): “[52] Zhaofan Qiu, Ting Yao, Chong-Wah Ngo, Xinmei Tian, and Tao Mei.”
Omni-sourced Webly-supervised Learning for Video Recognition (2020, unknown): “The Top-1 accuracy of the RGB stream improves by 2.7%.”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, unknown): “Method Pretrain #Frame GFLOPs K400 K600 Top-1 Top-5 Top-1 Top-5 LGD(Qiu et al., 2019) IN-1K 128×N/A N/A 79.”