C3D: Generic Features for Video Analysis

By Du Tran, Lubomir D. Bourdev, Rob Fergus, Lorenzo Torresani, Manohar Paluri. 2014

422 citations recorded by the source corpus.

Videos have become ubiquitous due to the ease of capturing and sharing via social platforms like Youtube, Facebook, Instagram, and others. The computer vision community has tried to tackle various video analysis problems independently. As a consequence, even though some really good hand-crafted features have been proposed there is a lack of generic features for video analysis. On the other hand, the image domain has progressed rapidly by using features from deep convolutional networks. These deep features are proving to be generic and perform well on variety of image tasks. In this work we propose Convolution 3D (C3D) feature, a generic spatio-temporal feature obtained by training a deep 3-dimensional convolutional network on a large annotated video dataset comprising objects, scenes, actions, and other frequently occurring concepts. We show that by using spatio-temporal convolutions the trained features encapsulate appearance and motion cues and perform well on different video classification tasks. C3D has three main advantages. First, it is generic: achieving state-of-the-art results on object recognition, scene classification, sport classification, and action similarity labeling in videos. Second, it is compact: obtaining better accuracy than best hand-crafted features and best deep image features with a lower dimensional feature descriptor. Third, it is efficient to compute: $91$ times faster than current hand-crafted features, and two orders of magnitude faster than current deep-learning based video classification methods.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Deep multi-scale video prediction beyond mean square error (2015, methodology): “Top performing algorithms for action recognition exploit the temporal information in a supervised way, such as the 3D convolutional network of Tran et al. (2015), or the spatio-temporal convolutional model of Simonyan & Zisserman (2014), which can require months of training, and heavily labeled…”
Action-Conditional Video Prediction using Deep Networks in Atari Games (2015, methodology): “hich is followed by a rectiﬁer nonlinearity [20], and a fully-connected layer at the end. This encoding can be viewed as early-fusion [12] (other types of fusions, e.g., late-fusion or 3D convolution [28] can also be applied to this architecture). Recurrent encoding takes one frame as an input for e”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, background): “More recently, 3D convolutions capture temporal information [27, 5, 35, 26].”
Unsupervised Learning of Video Representations using LSTMs (2015, background): “This has led to research in 3D convolutional nets (Ji et al., 2013; Tran et al., 2014), different temporal fusion strategies (Karpathy et al., 2014) and exploring different ways of presenting visual information to convolutional nets (Simonyan & Zisserman, 2014a).”
Describing Videos by Exploiting Temporal Structure (2015, methodology): “D CNN) which has recently been demonstrated to capture well the temporal dynamics in video clips [36, 17].”
Delving Deeper into Convolutional Networks for Learning Video Representations (2015, background): “…and task-speciﬁc representations (Wang et al., 2011; Sadanand & Corso, 2012), there is a growing interest in designing general video representations that could help solve tasks in video understanding such as human action recognition, video retrieval or video captionning (Tran et al., 2014).”
Recent advances in convolutional neural networks (2015, neutral): “[186] study the performance, efﬁciency, and effectiveness of this approach and show its strengths compared to other approaches.”
AssembleNet: Searching for Multi-Stream Neural Connectivity in Video Architectures (2019, neutral): “Full 3D space-time (i.e., XYT) convolutional layers as well as (2+1)D convolutional layers have been popularly used to represent videos [22, 3, 23, 30].”
Hierarchical Recurrent Neural Encoder for Video Representation with Application to Captioning (2015, methodology): “This approach, however, can only cope with short video clips of 16 frames or so [35].”
Video Paragraph Captioning Using Hierarchical Recurrent Neural Networks (2015, methodology): “trained C3D [44] (on the Sports-1M dataset [19]) for YouTubeClips.”
Jointly Modeling Embedding and Translation to Bridge Video and Language (2015, methodology): “D CNN of C3D [30] pre-trained on Sports-1M video dataset [15].”
Every Moment Counts: Dense Detailed Labeling of Actions in Complex Videos (2015, background): “[32] learn spatio-temporal filters in a deep network and apply them to a variety of human action understanding tasks.”
Audio-Visual Scene Analysis with Self-Supervised Multisensory Features (2018, background): “— a network design that is akin to video CNNs with inflated 3D convolutions [53,54].”
Orientation-boosted Voxel Nets for 3D Object Recognition (2016, methodology): “[14] use video frame stacks as a 3D signal to approach multiple video classification tasks using their 3D CNN, called C3D.”
Modeling Spatial-Temporal Clues in a Hybrid Deep Learning Framework for Video Classification (2015, neutral): “sidered as a signicant progress. UCF-101 CCV Donahue et al. [6] 82.9% Xu et al. [48] 60.3% Srivastava et al. [36] 84.3% Ye et al. [50] 64.0% Wang et al. [44] 85.9% Jhuo et al. [12] 64.0% Tran et al. [40] 86.7% Ma et al. [27] 63.4% Simonyan et al. [33] 88.0% Liu et al. [25] 68.2% Lan et al. [22] 89.1”
Revisiting the Effectiveness of Off-the-shelf Temporal Modeling Approaches for Large-scale Video Classification (2017, neutral): “(CNNs) [9, 13, 3, 17, 4] and/or recurrent networks [7, 15, 1] have achieved state-of-the-art results.”
Learning to Track for Spatio-Temporal Action Localization (2015, methodology): “Such approaches use 3D convolution on a stack of frames [16, 21, 38], apply recurrent neural network on per-frame features [4], or process images and optical ﬂows in two separate streams [35].”
Interpretable human action recognition in compressed domain (2017, background): “Widely used deep learning strategies for action recognition include [8, 9, 10, 11].”