Tube Convolutional Neural Network (T-CNN) for Action Detection in Videos

By Rui Hou, Chen Chen, Mubarak Shah. 2017

362 citations recorded by the source corpus.

Deep learning has been demonstrated to achieve excellent results for image classification and object detection. However, the impact of deep learning on video analysis has been limited due to complexity of video data and lack of annotations. Previous convolutional neural networks (CNN) based video action detection approaches usually consist of two major steps: frame-level action proposal generation and association of proposals across frames. Also, most of these methods employ two-stream CNN framework to handle spatial and temporal feature separately. In this paper, we propose an end-to-end deep network called Tube Convolutional Neural Network (T-CNN) for action detection in videos. The proposed architecture is a unified deep network that is able to recognize and localize action based on 3D convolution features. A video is first divided into equal length clips and next for each clip a set of tube proposals are generated based on 3D Convolutional Network (ConvNet) features. Finally, the tube proposals of different clips are linked together employing network flow and spatio-temporal action detection is performed using these linked video proposals. Extensive experiments on several video datasets demonstrate the superior performance of T-CNN for classifying and localizing actions in both trimmed and untrimmed videos compared to state-of-the-arts.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[52] proposed an end-to-end 3D CNN-based approach for action detection in videos.”
Long-Term Feature Banks for Detailed Video Understanding (2018, background): “The detected tubelets/boxes can then be optionally linked to form full action tubes [12, 17, 20, 32, 36, 40].”
Pattern Recognition (2015, positive): “At 0.2, our approach performs much better than [18,28,34,35,39] where with a large improvement by more than 25.2%, besides, it also outperforms the state-of-the-arts that rely on SSD [3] .”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, methodology): “Many recent CNNs for action classification [8, 52, 64, 60, 20], temporal action localization [66, 45, 65, 12, 70], and spatio-temporal action detection [19, 42, 41, 48, 4, 44, 62, 30, 74, 21, 26, 43] rely on pre-training a model on a large-scale dataset such as ImageNet or Kinetics and finetuning th”
STEm-Seg: Spatio-temporal Embeddings for Instance Segmentation in Videos (2020, methodology): “Other methods obtain video object proposals [25,29,35,42] based on cues such as actions [29,35] or image-level objects of interest [42,67].”
Learning Spatio-Temporal Representation With Local and Global Diffusion (2019, neutral): “[12] 78.”
A flexible model for training action localization with varying levels of supervision (2018, neutral): “The most recent methods [13, 15, 18, 35, 49] operate on multiple frames and leverage temporal information to determine actions.”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, methodology): “4% TCNN [16] 76.”