AMTnet: Action-Micro-Tube Regression by End-to-end Trainable Deep Architecture

By Suman Saha, Gurkirt Singh, Fabio Cuzzolin. 2017

73 citations recorded by the source corpus.

Dominant approaches to action detection can only provide sub-optimal solutions to the problem, as they rely on seeking frame-level detections, to later compose them into ‘action tubes’ in a post-processing step. With this paper we radically depart from current practice, and take a first step towards the design and implementation of a deep network architecture able to classify and regress whole video subsets, so providing a truly optimal solution of the action detection problem. In this work, in particular, we propose a novel deep net framework able to regress and classify 3D region proposals spanning two successive video frames, whose core is an evolution of classical region proposal networks (RPNs). As such, our 3D-RPN net is able to effectively encode the temporal aspect of actions by purely exploiting appearance, as opposed to methods which heavily rely on expensive flow maps. The proposed model is end-to-end trainable and can be jointly optimised for action localisation and classification in a single step. At test time the network predicts ‘micro-tubes’ encompassing two successive frames, which are linked up into complete action tubes via a new algorithm which exploits the temporal encoding learned by the network and cuts computation time by 50%. Promising results on the J-HMDB-21 and UCF-101 action detection datasets show that our model does outperform the state-of-the-art when relying purely on appearance.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[106] proposed a deep net framework capable of regressing and classifying 3D region proposals spanning two successive video frames.”
AVA: A Video Dataset of Spatio-Temporally Localized Atomic Visual Actions (2017, methodology): “Very recently, multi-frame approaches have emerged: Tubelets [41] jointly estimate localization and classification over several frames, T-CNN [16] use 3D convolutions to estimate short tubes, micro-tubes rely on two successive frames [33] and pose-guided 3D convolutions add pose to a two-stream appr”
Pattern Recognition (2015, unknown): “[3,17] cost much time on optical ﬂow computation and object detection.”
Rethinking Spatiotemporal Feature Learning For Video Understanding (2017, methodology): “UCF-101-24 is a subset of UCF-101 with 24 labels and 3207 videos; we use the cleaned bounding box annotations from [39].”
Why Can't I Dance in the Mall? Learning to Mitigate Scene Bias in Action Recognition (2019, methodology): “Many recent CNNs for action classification [8, 52, 64, 60, 20], temporal action localization [66, 45, 65, 12, 70], and spatio-temporal action detection [19, 42, 41, 48, 4, 44, 62, 30, 74, 21, 26, 43] rely on pre-training a model on a large-scale dataset such as ImageNet or Kinetics and finetuning th”
Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classification (2017, methodology): “UCF-101-24 is a subset of UCF-101 with 24 labels and 3207 videos; we use the cleaned bounding box annotations from [53].”
A flexible model for training action localization with varying levels of supervision (2018, unknown): “The most recent methods [13, 15, 18, 35, 49] operate on multiple frames and leverage temporal information to determine actions.”