Online Real-Time Multiple Spatiotemporal Action Localisation and Prediction

By Gurkirt Singh, Suman Saha, Michael Sapienza, Philip H. S. Torr, Fabio Cuzzolin. 2016

306 citations recorded by the source corpus.

We present a deep-learning framework for real-time multiple spatio-temporal (S/T) action localisation and classification. Current state-of-the-art approaches work offline, and are too slow to be useful in real-world settings. To overcome their limitations we introduce two major developments. Firstly, we adopt real-time SSD (Single Shot Multi-Box Detector) CNNs to regress and classify detection boxes in each video frame potentially containing an action of interest. Secondly, we design an original and efficient online algorithm to incrementally construct and label ‘action tubes’ from the SSD frame level detections. As a result, our system is not only capable of performing S/T detection in real time, but can also perform early action prediction in an online fashion. We achieve new state-of-the-art results in both S/T action localisation and early action prediction on the challenging UCF101-24 and J-HMDB-21 benchmarks, even when compared to the top offline competitors. To the best of our knowledge, ours is the first real-time (up to 40fps) system able to perform online S/T action localisation on the untrimmed videos of UCF101-24.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[119] designed an efficient online algorithm to incrementally construct and label ’action tubes’ from the SSD frame level detections, making it the first real-time (up to 40fps) system able to perform online S/T action localisation on the untrimmed videos.”
Video Action Transformer Network (2018, background): “Some recent work has looked at such ﬁne-grained video understanding [8, 17, 21, 37], but has largely been limited to small datasets like UCF-24 [37, 38] or JHMDB [19].”
Long-Term Feature Banks for Detailed Video Understanding (2018, background): “The detected tubelets/boxes can then be optionally linked to form full action tubes [12, 17, 20, 32, 36, 40].”
Multimodal Explanations by Predicting Counterfactuality in Videos (2018, methodology): “For the UCF101-24 dataset, we exploited the bounding box for the action detection provided in [25],”
Pattern Recognition (2015, unknown): “Compared to the traditional work ( i.e. dense trajectories) [2] , Convolutional Neural Network (CNN) based methods [3–5] get better performance in both accuracy and eﬃciency.”
Unified Graph Structured Models for Video Understanding (2021, background): “Consequently, whilst initial approaches to spatio-temporal action detection involved extending 2D object detectors [38, 46] temporally [26, 51, 1, 65], current leading methods [14, 61, 71] on the AVA dataset [17] all explicitly model relationships, with approaches which we show can be interpreted…”
Keeping Your Eye on the Ball: Trajectory Attention in Video Transformers (2021, background): “We see significant potential for using trajectory attention for tracking [28], temporal action localization [83, 81] and online action detection [86, 62], among other settings, and leave these as avenues for future work.”
LAEO-Net: Revisiting People Looking at Each Other in Videos (2019, methodology): “For constructing tracks, we use the online linking algorithm of [28], as it is robust to missed detections and can generate tracks spanning different temporal extents of the video.”
Part-Activated Deep Reinforcement Learning for Action Prediction (2018, background): “As an important branch of human activity analysis, predicting the activity of humans presents importance in a number of real-world applications, such as video detection [43], abnormal behavior detection [7, 40] and robot interaction [36].”
ECO: Efficient Convolutional Network for Online Video Understanding (2018, methodology): “In contrast to online action detection, which has been enabled recently [4], the approach provides not only fast reaction times, but also takes the longer term context into account.”
Rethinking the Faster R-CNN Architecture for Temporal Action Localization (2018, background): “Also note that besides temporal action localization, there also exists a large body of work on spatio-temporal action localization [19, 23, 40], which is beyond the scope of this paper.”
A flexible model for training action localization with varying levels of supervision (2018, methodology): “Currently, the dominant strategy [11, 31, 36, 37, 45] is to obtain perframe action detections and then to link them into continuous spatio-temporal tracks.”
Learning Spatio-Temporal Representation With Local and Global Diffusion (2019, unknown): “[35] 73.”