Rethinking Spatiotemporal Feature Learning For Video Understanding

By Saining Xie, Chen Sun, Jonathan Huang, Zhuowen Tu, Kevin P. Murphy. 2017

265 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

VideoBERT: A Joint Model for Video and Language Representation Learning (2019, methodology): “We use the pre-computed S3D features (same as the inputs to VideoBERT), applying average pooling over time, followed by a linear classiﬁer.”
MERLOT: Multimodal Neural Script Knowledge Models (2021, methodology): “, S3D [104]) we adopt a sparse sampling approach, collecting a sequence of still frames from each video clip instead (we generally use 5, but details per-task are in Appendix D.”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, methodology): “The AvgPool baseline averages S3D features up to one second before the prediction starts and learns a linear classifier for action prediction.”
SlowFast Networks for Video Recognition (2018, methodology): “There are also related methods focusing on long-term filtering and pooling using temporal strides [52, 13, 55, 62], as well as decomposing the convolutions into separate 2D spatial and 1D temporal filters [12, 50, 61, 39].”
Multiscale Vision Transformers (2021, background): “Incorporating downsampling, shift invariance, and shared weights, ConvNets are de-facto standard backbones for computer vision tasks for image [65, 62, 88, 90, 46, 12, 15, 34, 93, 81, 41] and video [87, 31, 11, 79, 69, 106, 96, 30, 105, 35, 29, 117, 57].”
Large-Scale Weakly-Supervised Pre-Training for Video Action Recognition (2019, methodology): “Compared to other approaches that use only RGB as input [68], our approach yields a boost of 3.”
Long-Term Feature Banks for Detailed Video Understanding (2018, background): “Deep networks are the dominant approach for video understanding [5, 21, 33, 39, 46–48, 50, 51, 56].”
MViTv2: Improved Multiscale Vision Transformers for Classification and Detection (2021, background): “CNNs serve as the primary backbones for computer vision tasks, including image recognition [10,15,34,39,46,48,62, 67,69,71], object detection [6,29,38,53,63,93] and video recognition [8,22,23,25,28,42,51,61,66,75,79,84,92].”
Graph-Based Global Reasoning Networks (2018, unknown): “S3D-G [35] Inception-v1 64 71.”
Self-supervised Co-training for Video Representation Learning (2020, unknown): “We choose the S3D [65] architecture as the feature extractor for all experiments.”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, methodology): “Other works separately factor convolutions into separate 2D spatial and 1D temporal filters [24, 60, 64, 48].”
Learning Video Representations using Contrastive Bidirectional Transformer (2019, methodology): “on Most of the recent efforts on learning deep representations for videos have been focusing on capturing the appearance and local motion pattern from short video clips [22, 13, 11], in an supervised [26, 5, 35] or self-supervised [15, 31] fashion. However, to understand longer videos, we need power”
Multi-Fiber Networks for Video Recognition (2018, methodology): “Recently, S3D [8] and R(2+1)D [2] are proposed to use one 1 × 3 × 3 convolution layer followed by another 3 × 1 × 1 convolutional layer to approximate a full-rank 3D kernel to reduce the computations of a full-rank 3 × 3 × 3 convolutional layer while achieving better precision.”
End-to-end Generative Pretraining for Multimodal Video Captioning (2022, methodology): “Early works implemented the visual encoder as a 2D CNN (either frozen or finetuned) applied to video frames, which was then naturally extended to 3D CNNs [6,53], to better capture motion dynamics, with temporal aggregation over the entire video typically performed using attention strategies [9].”
Prompting Visual-Language Models for Efficient Video Understanding (2021, background): “In the last decade, research on developing effective architectures has gone through rapid developments, from two-stream networks [15, 61, 66] to more recent single stream RGB networks [2, 8, 13, 14, 65, 70].”