SlowFast Networks for Video Recognition

By Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, Kaiming He. 2018

4,284 citations recorded by the source corpus.

We present SlowFast networks for video recognition. Our model involves (i) a Slow pathway, operating at low frame rate, to capture spatial semantics, and (ii) a Fast pathway, operating at high frame rate, to capture motion at fine temporal resolution. The Fast pathway can be made very lightweight by reducing its channel capacity, yet can learn useful temporal information for video recognition. Our models achieve strong performance for both action classification and detection in video, and large improvements are pin-pointed as contributions by our SlowFast concept. We report state-of-the-art accuracy on major video recognition benchmarks, Kinetics, Charades and AVA. Code has been made available at: https://github.com/facebookresearch/SlowFast.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

Late Temporal Modeling in 3D CNN Architectures with BERT for Action Recognition (2020, methodology): “However, SGD is preferred for 3D CNN architectures [12,1,7,28,3].”
Self-Supervised MultiModal Versatile Networks (2020, neutral): “Supervised [21, 42, 67, 74, 90] 96.”
Multiview Transformers for Video Recognition (2022, methodology): “SlowFast [23] proposes a twostream CNN architecture that takes frames sampled at two different frame rates.”
VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modeling (2021, methodology): “Different from MRM and MFM, which rely on distilled visual categories or features from a well-supervised visual backbone [41, 45], we present Masked Visual-token Modeling (MVM) to perform masked visual modeling in a self-reconstruction scenario.”
Hero: Hierarchical Encoder for Video+Language Omni-representation Pre-training (2020, methodology): “, 2009) and SlowFast (Feichtenhofer et al., 2019) pre-trained on Kinetics (Kay et al.”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “With the SlowFast network [11], we show that by adding hand masks into training, the top-1 classification accuracy of “verbs/nouns” boosts from 23.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): ““Features” refers to precomputed SlowFast [70] video features.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “ripted or curated (YouTube) videos. Evaluation Metrics. We report Top-1/5 Accuracy on Val and Test sets. Baselines and Results. In Table 3, we report results of ve state-of-the-art recognition models [47,48,49,50,51] in addition to a random chance baseline. We use the Train set to report on Val, opt”
The Efficiency Misnomer (2021, methodology): “Despite this, FLOPs has been used as the most common cost indicator in many research papers, especially in the recent computer vision literature, to quantify model efﬁciency (Szegedy et al., 2015; He et al., 2016; Tan and Le, 2019; Feichtenhofer et al., 2019; Fan et al., 2021).”
Less is More: CLIPBERT for Video-and-Language Learning via Sparse Sampling (2021, methodology): “Note that HERO uses SlowFast [14] features extracted from full-length videos at a very dense frame rate of 21 FPS (i.e., on average 310 frames per video for MSRVTT), while CLIPBERT 4×1 uses only 4 randomly sampled frames.”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, positive): “This could be done by preparing actual videos with different temporal resolutions as in [6] or by using temporally ‘dilated’ convolutions as we introduce here.”
Class-Incremental Learning for Action Recognition in Videos (2021, neutral): “So, the proposed class-incremental learning framework employs a frame-based video representation method—Temporal Shift Module (TSM) [21], and reduces computational cost for training signiﬁcantly compared to 3D CNNs based on video volumes [2, 8, 9, 36, 37] and their variations [40, 47].”
VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Text (2021, methodology): “We uniformly sample 4 clips to cover the entire 10 seconds of the video and apply a standard 3-crop evaluation following [34].”
Is Space-Time Attention All You Need for Video Understanding? (2021, methodology): “In contrast, SlowFast 8x8 R50 has a larger inference cost ( 1 .”
Multiscale Vision Transformers (2021, methodology): “In Table 16, we analyze the speed/accuracy trade-off of our MViT models, along with their counterparts vision transformer (ViT [25]) and ConvNets (SlowFast 8×8 R50, SlowFast 8×8 R101 [30], & X3D-L [29]).”
ViViT: A Video Vision Transformer (2021, methodology): “Inference The input to our network is a video clip of 32 frames using a stride of 2, unless otherwise mentioned, similar to [21, 20].”
Video Representation Learning by Dense Predictive Coding (2019, methodology): “Following the convention of [5] there are four residual blocks in ResNet architecture, namely res2, res3, res4 and res5, and only expand the convolutional kernels in res4 and res5 to be 3D ones.”
CrossViT: Cross-Attention Multi-Scale Vision Transformer for Image Classification (2021, background): “SlowFast Networks [13] rely on a similar two-branch model, but each branch encodes different frame rates, as opposed to frames with different spatial resolutions.”
VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation (2021, methodology): “We use ResNet-152 [22] pre-trained on ImageNet [14] to extract 2D features, and use SlowFast [19] pre-trained on Kinetics [27] to extract 3D features.”
VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding (2021, methodology): “Notably, using a frozen video backbone (fθCNN) enables us to go beyond short-term visual input (typical video CNNs (Xie et al., 2018; Feichtenhofer et al., 2019) only capture temporal windows of ∼3 seconds), and allows us to model long-term visual-textual correspondences spanning ∼32 seconds.”