Online Action Detection

By Roeland De Geest, Efstratios Gavves, Amir Ghodrati, Zhenyang Li, Cees G. M. Snoek, Tinne Tuytelaars. 2016

189 citations recorded by the source corpus.

In online action detection, the goal is to detect the start of an action in a video stream as soon as it happens. For instance, if a child is chasing a ball, an autonomous car should recognize what is going on and respond immediately. This is a very challenging problem for four reasons. First, only partial actions are observed. Second, there is a large variability in negative data. Third, the start of the action is unknown, so it is unclear over what time window the information should be integrated. Finally, in real world data, large within-class variability exists. This problem has been addressed before, but only to some extent. Our contributions to online action detection are threefold. First, we introduce a realistic dataset composed of 27 episodes from 6 popular TV series. The dataset spans over 16 h of footage annotated with 30 action classes, totaling 6,231 action instances. Second, we analyze and compare various baseline methods, showing this is a challenging problem for which none of the methods provides a good solution. Third, we analyze the change in performance when there is a variation in viewpoint, occlusion, truncation, etc. We introduce an evaluation protocol for fair comparison. The dataset, the baselines and the models will all be made publicly available to encourage (much needed) further research on online action detection on realistic data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “s Instances Hours Avg. Action Length Avg. Instances per Video Avg. Classes per Video Labelled Frames Overlapping Segments TV Human Interactions [54] 7 3 7 4 300 0.3 3.9s 1.0 1.0 100.0% 0.0% TV Series [55] 7 3 7 30 6,231 16.0 1.9s 230.7 23.5 N/A 21.9% THUMOS 14 [56] 3 3 3 101 6,310 30.0 4.2s 15.4 1.1”
RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “[37, 39, 38, 33, 36]) have encouraged research and challenges/contests in motion recognition in this domain.”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “De Geest et al. [6] released a new dataset for online action detection and benchmarked several baseline methods to address the task.”
Transition Forests: Learning Discriminative Temporal Transitions for Action Recognition and Detection (2016, background): “Detecting actions on streaming data [7] has been less explored than recognizing segmented sequences, while being more interesting in real scenarios.”
Cascaded Boundary Regression for Temporal Action Detection (2017, methodology): “We first introduce the datasets and the experiment setup, then discuss the experimental results on TVSeries [2].”
What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019, background): “, to anticipate a pedestrian crossing the street from an autonomous vehicle [9]) or even before it starts (e.”
Online action detection (2017, unknown): “Online action detection is a very difficult task, and there is no work specifically addressed on this problem, except [2] has proposed some very basic baseline method.”