Spatiotemporal Multiplier Networks for Video Action Recognition

By Christoph Feichtenhofer, Axel Pinz, Richard P. Wildes. 2017

615 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, positive): “Previously, researchers approached this challenge by hand-designing a two-stream model taking two different types of inputs (e.g., RGB pixels and optical ﬂow), obtaining successful results [20, 9, 7, 8, 6].”
Video Representation Learning Using Discriminative Pooling (2018, methodology): “Successful methods following this approach are the two-stream models and their extensions [11, 12, 18, 20, 41].”
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? (2017, methodology): “Furthermore, we can also see that ResNeXt-101 (64f), which utilize larger inputs described in previous section, slightly outperformed ST Multiplier Net [7] and TSN [29], which utilize more complex two-stream architectures.”
VideoLSTM convolves, attends and flows for action recognition (2016, positive): “, Simonyan and Zisserman (2014), Feichtenhofer et al. (2016b) and Wang et al. (2016). Different from these modern architectures, which all learn a global video representation without considering local spatial structures over time, we prefer to embed the spatio-temporal nature of the action into our ”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, background): “Feichtenhofer et al. [24], [25], [26] studied approaches to fuse predictions performed by the motion and appearance streams of a 2SCNN to improve recognition.”
Learnable pooling with Context Gating for video classification (2017, methodology): “More recent methods extract deep convolutional neural network activations computed from individual frames or blocks of frames using spatial [6], [25], [26], [27] or spatio-temporal [5], [10], [11], [12], [13], [14] convolutions.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “Some works explore the use of 2D networks for video recognition by processing videos in smaller segments followed by late fusion [31, 15, 74, 68, 20, 58, 36, 39, 69, 75, 76].”
Spatiotemporal-Textual Co-Attention Network for Video Question Answering (2019, background): “…top layers of deep convolutional neural network provide semantic representation of visual content and have been shown as effective features in many vision tasks, such as in References [5, 10, 11, 13, 26, 48], e.g., object detection [22], image captioning [23, 28], and video action recognition [7].”
VA-RED2: Video Adaptive Redundancy Reduction (2021, background): “Several works have been proposed that add decision branches to different layers of CNNs to learn whether to exit the network for faster inference (Yu et al., 2018; Figurnov et al., 2017; McGill & Perona, 2017; Teerapittayanon et al., 2016) Wang et al. (Wang et al., 2018) propose to skip convolutiona”
Optical Flow Guided Feature: A Fast and Robust Motion Representation for Video Action Recognition (2017, methodology): “As a significant breakthrough in action recognition, TwoStream based frameworks used the deep CNN to learn from the hand-craft motion features like optical flow and iDT [29, 41, 50, 43, 9, 47, 5, 35, 11, 12].”
DistInit: Learning Video Representations Without a Single Labeled Video (2019, methodology): “Similar ideas of inflating 2D models to 3D have been proposed previously for Inception-style architectures [4], along with initialization techniques from corresponding 2D models [4, 8, 9].”
Recognizing Human Actions as the Evolution of Pose Estimation Maps (2018, background): “Based on this network, the relationships between the spatial and temporal structures were further explored [11, 45].”
Multi-Fiber Networks for Video Recognition (2018, neutral): “Method FLOPs +OF UCF-101 HMDB51 ResNet-50 [37] 3.”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, background): “Since then, many video classiﬁcation methods follow the same multi-stream 2D CNN design, and have made improvements in terms of new representations [25,26], different backbone architecture [27,28,29,17], fusion of the streams [30,31,32,33] and exploiting richer temporal structures [34,35,36].”
TEA: Temporal Excitation and Aggregation for Action Recognition (2020, background): “[9, 8] explored different mid-level combination strategies to fuse the features of two streams.”
HACS: Human Action Clips and Segments Dataset for Recognition and Temporal Localization (2017, methodology): “By using the off-the-shelf I3D and R(2+1)D models, and leveraging a large-scale clip dataset, our approach outperforms other methods [54, 16, 58, 8, 53] on all 3 benchmarks.”
STM: SpatioTemporal and Motion Encoding for Action Recognition (2019, methodology): “The first type is based on twostream neural networks [10, 33, 36, 9], which consists of an RGB stream with RGB frames as input and a flow stream with optical flow as input.”
Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, methodology): “In action classification, most of the state-of-the-art methods [37, 30, 49, 7, 14] rely on a two-stream architecture, which parallelly processes two types of input—RGB frames and pre-computed optical flow—and later fuses their features to generate the final classification scores.”