Long-Term Temporal Convolutions for Action Recognition

By Gul Varol, Ivan Laptev, Cordelia Schmid. 2016

951 citations recorded by the source corpus.

Typical human actions last several seconds and exhibit characteristic spatio-temporal structure. Recent methods attempt to capture this structure and learn action representations with convolutional neural networks. Such representations, however, are typically learned at the level of a few video frames failing to model actions at their full temporal extent. In this work we learn video representations using neural networks with long-term temporal convolutions (LTC). We demonstrate that LTC-CNN models with increased temporal extents improve the accuracy of action recognition. We also study the impact of different low-level representations, such as raw values of video pixels and optical flow vector fields and demonstrate the importance of high-quality optical flow estimation for learning accurate action models. We report state-of-the-art results on two challenging benchmarks for human action recognition UCF101 (92.7%) and HMDB51 (67.2%).

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Spatio-Temporal Features with 3D Residual Networks for Action Recognition (2017, background): “High spatial resolutions and long temporal durations improve recognition accuracy [19].”
Skeleton-Based Online Action Prediction Using Scale Selection Network (2019, background): “Convolutional networks [74] have proven their superior strength in modeling the time series data [75], [76], [77].”
2D/3D Pose Estimation and Action Recognition Using Multitask Deep Learning (2018, background): “scores [8, 10, 53], but they involve high number of parameters, require an elevated amount of memory for training, and cannot efficiently benefit from the abundant still images for training.”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[133] investigated the learning of long-term video representations and proposed the Long-term Temporal Convolutions (LTC) at the expense of decreasing spatial resolution to keep the complexity of networks tractable.”
Skeleton-Based Action Recognition Using Spatio-Temporal LSTM Network with Trust Gates (2017, background): “HUMAN action recognition is a fast developing research area due to its wide applications in intelligent surveillance, human-computer interaction, robotics, and so on [1], [2].”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, background): “Actions can span various time intervals, making good use of videos and their temporal context is a prerequisite for solving the task to its full extent [42, 41].”
A Closer Look at Spatiotemporal Convolutions for Action Recognition (2017, background): “This is consistent with the findings in [37].”
Learning a Text-Video Embedding from Incomplete and Heterogeneous Data (2018, background): “Most prior works [12], [31], [33], [37], [38] address the problem of appearance and motion fusion for video representation.”
Can Spatiotemporal 3D CNNs Retrace the History of 2D CNNs and ImageNet? (2017, background): “showed that expanding the temporal length of inputs for C3D improves recognition performance [25].”
Temporal Segment Networks for Action Recognition in Videos (2017, methodology): “Compared with those regular sampling methods [4], [24],”
Long-Term Feature Banks for Detailed Video Understanding (2018, background): “Deep networks are the dominant approach for video understanding [5, 21, 33, 39, 46–48, 50, 51, 56].”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, background): “Such findings hint that it is possible to predict when actions begin and end, and have inspired hundreds of Computer Vision datasets, models, and algorithms to study tasks like action recognition [19, 37, 71, 80, 81, 82].”
Learnable pooling with Context Gating for video classification (2017, methodology): “Convolutional neural networks can be also applied separately on the appearance channel and the pre-computed motion field channel resulting in the, so called, two-stream representations [6], [11], [14], [26], [28].”
ECO: Efficient Convolutional Network for Online Video Understanding (2018, methodology): “To cope with the problem of partial observation, some methods increased the temporal resolution of the sliding window [16,17].”
Predictive-Corrective Networks for Action Detection (2017, background): “rectly process spatio-temporal volumes exist [42, 43].”
Deep Temporal Linear Encoding Networks (2016, neutral): “…Method UCF101 HMDB51 DT+MVSM [2] 83.5 55.9 iDT+FV [34] 85.9 57.2 Two Stream [24] 88.0 59.4 VideoDarwin [8] − 63.7 C3D [32] 82.3 56.8 Two Stream+LSTM [40] 88.6 − F ST CV (SCI fusion) [29] 88.1 59.1 TDD+FV [36] 90.3 63.2 LTC [33] 91.7 64.8 KVMF [43] 93.1 63.3 TSN [37] 94.0 68.5 3DConv+3DPool [7] 93.”
Self-Supervised Visual Feature Learning With Deep Neural Networks: A Survey (2019, background): “to train 3DConvNets and achieved better performance than C3D [67], [68].”
ActionVLAD: Learning Spatio-Temporal Aggregation for Action Classification (2017, background): “This type of representation, though shallow, is still relevant today and is in fact part of the existing stateof-the-art systems [15, 51, 55].”
Hidden Two-Stream Convolutional Networks for Action Recognition (2017, methodology): “In the end, we fuse the two streams’ scores with a spatial to temporal stream ratio of 1:1.5.”