2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition

By Hengduo Li, Zuxuan Wu, Abhinav Shrivastava, Larry S. Davis. 2020

35 citations recorded by the source corpus.

3D convolutional networks are prevalent for video recognition. While achieving excellent recognition performance on standard benchmarks, they operate on a sequence of frames with 3D convolutions and thus are computationally demanding. Exploiting large variations among different videos, we introduce Ada3D, a conditional computation framework that learns instance-specific 3D usage policies to determine frames and convolution layers to be used in a 3D network. These policies are derived with a two-head lightweight selection network conditioned on each input video clip. Then, only frames and convolutions that are selected by the selection network are used in the 3D model to generate predictions. The selection network is optimized with policy gradient methods to maximize a reward that encourages making correct predictions with limited computation. We conduct experiments on three video recognition benchmarks and demonstrate that our method achieves similar accuracies to state-of-the-art 3D models while requiring 20% − 50% less computation across different datasets. We also show that learned policies are transferable and Ada3D is compatible to different backbones and modern clip selection approaches. Our qualitative analysis indicates that our method allocates fewer 3D convolutions and frames for "static" inputs, yet uses more for motion-intensive clips.

Subfields: Computer Vision and Pattern Recognition, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Shiji Song
2. Yulin Wang
3. Gao Huang
4. Zuxuan Wu
5. Yu-Gang Jiang
6. Hengduo Li
7. Ser-Nam Lim
8. Yizeng Han

In Efficiency & Compression

1. Shiji Song
2. Gao Huang
3. Yulin Wang
4. Le Yang
5. Yizeng Han
6. Honghui Wang
7. Yu-Gang Jiang
8. Hengduo Li

Citations in the corpus

• Dynamic Neural Networks: A Survey (2021, background): “Although it has been observed that a low resolution might be sufficient for recognizing most ”easy” samples [27], conventional CNNs mostly process all the inputs with the same resolution, inducing considerable redundancy.”
• Adaptive Focus for Efficient Video Recognition (2021, background): “Recent research efforts have been made towards improving the efficiency of video recognition, via designing light-weighted architectures [38, 46, 32, 37, 54, 27] or perform dynamic computation on a per-video basis [49, 45, 25, 52, 26, 30].”
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition (2021, methodology): “In particular, existing methods for CNNs have explored altering input samples [28, 29, 39, 50, 51, 55], skipping network layers [10, 18, 41, 45, 52] and channels [1, 21], early exiting with a multi-classifier structure [2,17,19], to name a few.”