Video Classification With Channel-Separated Convolutional Networks

By Du Tran, Heng Wang, Lorenzo Torresani, Matt Feiszli. 2019

655 citations recorded by the source corpus.

Group convolution has been shown to offer great computational savings in various 2D convolutional architectures for image classification. It is natural to ask: 1) if group convolution can help to alleviate the high computational cost of video classification networks; 2) what factors matter the most in 3D group convolutional networks; and 3) what are good computation/accuracy trade-offs with 3D group convolutional networks. This paper studies the effects of different design choices in 3D group convolutional networks for video classification. We empirically demonstrate that the amount of channel interactions plays an important role in the accuracy of 3D group convolutional networks. Our experiments suggest two main findings. First, it is a good practice to factorize 3D convolutions by separating channel interactions and spatiotemporal interactions as this leads to improved accuracy and lower computational cost. Second, 3D channel-separated convolutions provide a form of regularization, yielding lower training accuracy but higher test accuracy compared to 3D convolutions. These two empirical findings lead us to design an architecture -- Channel-Separated Convolutional Network (CSN) -- which is simple, efficient, yet accurate. On Sports1M and Kinetics, our CSNs are comparable with or better than the state-of-the-art while being 2-3 times more efficient.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, background): “Another important 3D CNN architecture is Channel-Separated Convolutional Networks (CSN) [27] which separates the channel interactions and spatio-temporal interactions which can be thought as the 3D CNN version of depth-wise separable convolution [14].”
Multiview Transformers for Video Recognition (2022, neutral): “Since the release of the Kinetics dataset [33], 3D CNNs [10, 24, 69] have gained popularity, and many variants [22, 63, 70, 71, 79] have been developed to improve the speed and accuracy.”
Class-Incremental Learning for Action Recognition in Videos (2021, positive): “However, 3D CNN-based models are computationally expensive since they involve a large number of parameters to learn.”
Is Space-Time Attention All You Need for Video Understanding? (2021, neutral): “…93.4 N/A Oct-I3D+NL (Chen et al., 2019) 75.7 N/A 0.84 D3D (Stroud et al., 2020) 75.9 N/A N/A I3D+NL (Wang et al., 2018b) 77.7 93.3 10.8 ip-CSN-152 (Tran et al., 2019) 77.8 92.8 3.2 CorrNet (Wang et al., 2020a) 79.2 N/A 6.7 LGD-3D-101 (Qiu et al., 2019) 79.4 94.4 N/A SlowFast (Feichtenhofer et…”
Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [70, 67, 94, 96, 51, 15, 18, 39, 99, 87, 46] and video [93, 36, 14, 85, 74, 112, 102, 34, 111, 40, 33, 123, 62].”
Unidentified Video Objects: A Benchmark for Dense, Open-World Segmentation (2021, background): “Current video understanding approaches [36, 5, 38, 12, 37] cannot scale well to long videos due to GPU memory constraints and are not designed for complex prediction tasks beyond classification and detection.”
When Video Classification Meets Incremental Classes (2021, background): “With the rise of deep learning, CNNs dominated the video understanding literature recently [3, 21, 27, 30, 36, 44, 54, 56], instead of hand-crafted features [1, 26, 35, 46, 47].”
RubiksNet: Learnable 3D-Shift for Efficient Video Action Recognition (2020, neutral): “Tran et al. [24] factorizes 3D group convolutions while preserving channel interactions.”
Revisiting Skeleton-based Action Recognition (2021, background): “From then on, many advanced 3DCNN architectures [33, 11, 32, 10] were proposed by the action recognition community, which outperforms I3D both in recognition performance and efficiency.”
SCSampler: Sampling Salient Clips From Video for Efficient Action Recognition (2019, methodology): “Note that [43] reports an accuracy of 79.”
Anticipative Video Transformer (2021, methodology): “Finally, we experiment with irCSN-152 [84] pretrained using IG65M [27] and finetuned on EK55, and find it to outperform all methods by a significant margin on top-1/5.”
UniFormer: Unified Transformer for Efficient Spatiotemporal Representation Learning (2022, methodology): “2, our UniFormer block in the shallow layers can be interpreted as a transformer-style spatiotemporal MobileNet block (Tran et al., 2019) with extra FFN.”
FrameExit: Conditional Early Exiting for Efficient Video Recognition (2021, methodology): “This can be obtained by decomposing 3D filters into separate 2D spatial and 1D temporal filters [43, 36, 53], extending efficient 2D architectures to their 3D counterparts [42, 28], using shifting operations [31, 9], or exploring neural architecture search for videos [35, 11].”