Action recognition with trajectory-pooled deep-convolutional descriptors

By Limin Wang, Yu Qiao, Xiaoou Tang. 2015

1,199 citations recorded by the source corpus.

Visual features are of vital importance for human action understanding in videos. This paper presents a new video representation, called trajectory-pooled deep-convolutional descriptor (TDD), which shares the merits of both hand-crafted features [31] and deep-learned features [24]. Specifically, we utilize deep architectures to learn discriminative convolutional feature maps, and conduct trajectory-constrained pooling to aggregate these convolutional features into effective descriptors. To enhance the robustness of TDDs, we design two normalization methods to transform convolutional feature maps, namely spatiotemporal normalization and channel normalization. The advantages of our features come from (i) TDDs are automatically learned and contain high discriminative capacity compared with those hand-crafted features; (ii) TDDs take account of the intrinsic characteristics of temporal dimension and introduce the strategies of trajectory-constrained sampling and pooling for aggregating deep-learned features. We conduct experiments on two challenging datasets: HMD-B51 and UCF101. Experimental results show that TDDs outperform previous hand-crafted features [31] and deep-learned features [24]. Our method also achieves superior performance to the state of the art on these datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Endocrinology, Diabetes and Metabolism

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Endocrinology, Diabetes and Metabolism

Citations in the corpus

Non-local Neural Networks (2017, background): “In addition to endto-end modeling on raw video inputs, it has been found that optical flow [45] and trajectories [50, 51] can be helpful.”
End-to-End Learning of Visual Representations From Uncurated Instructional Videos (2019, methodology): “Most existing methods [23, 34, 38, 39] rely on supervised learning and, therefore, require large-scale, diverse and representative action datasets for training.”
Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset (2017, neutral): “…is even larger, although Model UCF-101 HMDB-51 Two-Stream [25] 88.0 59.4 IDT [30] 86.4 61.7 Dynamic Image Networks + IDT [2] 89.1 65.2 TDD + IDT [31] 91.5 65.9 Two-Stream Fusion + IDT [8] 93.5 69.2 Temporal Segment Networks [32] 94.2 69.4 ST-ResNet + IDT [7] 94 C3D is trained on more videos, 1M…”
Temporal Segment Networks: Towards Good Practices for Deep Action Recognition (2016, methodology): “Video-based action recognition has drawn a significant amount of attention from the academic community [1,2,3,4,5,6], owing to its applications in many areas like security and behavior analysis.”
Delving Deeper into Convolutional Networks for Learning Video Representations (2015, methodology): “Our model is on part with Wang et al. (2015b) that obtain state-of-art results using both RGB and ﬂow streams which obtains 90 .”
Spatiotemporal Residual Networks for Video Action Recognition (2016, neutral): “wo-Stream (VGG16) [1,31] 91.4% 58.5% Transformations[31] 92.4% 62.0% Two-Stream Fusion[5] 92.5% 65.4% ST-ResNet* 93.4% 66.4% Method UCF101 HMDB51 IDT [29] 86.4% 61.7% C3D + IDT [26] 90.4% - TDD + IDT [30] 91.5% 65.9% Dynamic Image Networks + IDT [2] 89.1% 65.2% Two-Stream Fusion[5] 93.5% 69.2% ST-Re”
Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos (2016, methodology): “[76] combined the advantages of hand-crafted trajectory-based features and deep convnet learning based methods by applying [56]’s network along the motion trajectories of input videos.”
Video Representation Learning Using Discriminative Pooling (2018, background): “For example, in the popular two-stream model [13, 41, 42, 53, 55], the CNNs are”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, methodology): “[145] extended the two-stream networks by integrating improved trajectories [141], where trajectory-constrained sampling and pooling are used to encode deep features learned from deep CNN architecture, into effective descriptors.”
Chained Multi-stream Networks Exploiting Pose, Motion, and Appearance for Action Classification and Detection (2017, neutral): “Several works employed deep architectures for classification [26, 41, 45, 34].”
Scene Flow to Action Map: A New Representation for RGB-D Based Action Recognition with Convolutional Neural Networks (2017, background): “Two kinds of motion representations can be identified: Lagrangian motion [48, 18, 31, 50, 30, 56] and Eulerian motion [2, 25, 60, 51, 52, 1].”
Unsupervised Learning of Long-Term Motion Dynamics for Videos (2017, background): “However, unlike image classification, the benefit of using deep neural networks over traditional hand crafted features is not very evident, since supervised training of deep networks requires a lot of data, whilst the current RGB activity recognition datasets are still too small.”
Towards Good Practices for Very Deep Two-Stream ConvNets (2015, methodology): “For the extraction of optical flow fields, we follow the work of TDD [19] and choose the TVL1 optical flow algorithm [21].”
Convolutional Two-Stream Network Fusion for Video Action Recognition (2016, positive): “Indeed the current state of the art performance [30, 34] on standard benchmarks such as UCF101 [24] and HMDB51 [13] is achieved by a combination of ConvNets and a Fisher Vector encoding [20] of hand-crafted features (such as HOF [14] over dense trajectories [33]).”
Actions ~ Transformations (2015, methodology): “Learned representations using deep learning have recently produced state of the art results in action recognition [13, 17, 32, 44, 40, 41, 23, 8, 49].”
Long-Term Temporal Convolutions for Action Recognition (2016, background): “Extensions of CNN representations to action recognition in video have been proposed in several recent works [6,12,13,14,15, 16, 22, 23].”
Temporal Convolutional Networks for Action Segmentation and Detection (2016, background): “[29], which combines their EgoNet features with TDD [18].”