X3D: Expanding Architectures for Efficient Video Recognition

By Christoph Feichtenhofer. 2020

1,353 citations recorded by the source corpus.

This paper presents X3D, a family of efficient video networks that progressively expand a tiny 2D image classification architecture along multiple network axes, in space, time, width and depth. Inspired by feature selection methods in machine learning, a simple stepwise network expansion approach is employed that expands a single axis in each step, such that good accuracy to complexity trade-off is achieved. To expand X3D to a specific target complexity, we perform progressive forward expansion followed by backward contraction. X3D achieves state-of-the-art performance while requiring 4.8x and 5.5x fewer multiply-adds and parameters for similar accuracy as previous work. Our most surprising finding is that networks with high spatiotemporal resolution can perform well, while being extremely light in terms of network width and parameters. We report competitive accuracy at unprecedented efficiency on video classification and detection benchmarks. Code is available at: https://github.com/facebookresearch/SlowFast.

Subfields: Computational Mechanics, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computational Mechanics

In Computer Vision and Pattern Recognition

Citations in the corpus

Multiview Transformers for Video Recognition (2022, neutral): “Since the release of the Kinetics dataset [33], 3D CNNs [10, 24, 69] have gained popularity, and many variants [22, 63, 70, 71, 79] have been developed to improve the speed and accuracy.”
Class-Incremental Learning for Action Recognition in Videos (2021, neutral): “So, the proposed class-incremental learning framework employs a frame-based video representation method—Temporal Shift Module (TSM) [21], and reduces computational cost for training signiﬁcantly compared to 3D CNNs based on video volumes [2, 8, 9, 36, 37] and their variations [40, 47].”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “Compared to CNNs, Transformers impose less restrictive inductive biases.”
ViViT: A Video Vision Transformer (2021, methodology): “Inference The input to our network is a video clip of 32 frames using a stride of 2, unless otherwise mentioned, similar to [21, 20].”
Revisiting ResNets: Improved Training and Scaling Strategies (2021, neutral): “Similarly to our study on image classiﬁ-cation (Table 1), we ﬁnd that most of the improvement can (Feichtenhofer, 2020).”
When Video Classification Meets Incremental Classes (2021, background): “In the next few years, 3D CNNs advanced quickly and became top performers on almost every benchmark dataset [10, 11, 51].”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, methodology): “Improving the efficiency of video models has gained increased attention [19, 64, 20, 18, 40, 17, 3, 11, 37, 48].”
Spatiotemporal Contrastive Video Representation Learning (2020, background): “Similarly, state-of-the-art neural networks for video understanding [39, 3, 19, 10, 9] extend 2D convolutional neural networks [21, 23] for images along the temporal dimension.”
VideoMamba: State Space Model for Efficient Video Understanding (2024, neutral): “Initially, 3D CNNs [9, 18, 77, 78] expanded traditional 2D CNNs to capture spatiotemporal information.”
2D or not 2D? Adaptive 3D Convolution Selection for Efficient Video Recognition (2020, methodology): “For inference, we follow the common practice [9, 8, 39] and uniformly sample 10 clips with a spatial size of 256 × 256 from a testing video.”
TokenLearner: What Can 8 Learned Tokens Do for Images and Videos? (2021, methodology): “Speciﬁcally, we modiﬁed X3D to be more computationally efﬁcient by (1) replacing its 3D XYT convolutional layers with a pair of 2D conv. layer and 1D conv. layer, and (2) removing Squeeze-and-Excitation layers [29] and swish activations.”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “Current algorithms can reliably recognize the action the subject is performing in many unconstrained settings from third person viewpoints [9, 22, 23, 24, 71, 89].”
VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Training (2022, methodology): “The supervised learning methods (Wang et al., 2019; Tran et al., 2018; Feichtenhofer et al., 2019; Feichtenhofer, 2020; Bertasius et al., 2021) usually depend on the image backbones.”