Timeception for Complex Action Recognition

By Noureldien Hussein, Efstratios Gavves, Arnold M. W. Smeulders. 2018

237 citations recorded by the source corpus.

This paper focuses on the temporal aspect for recognizing human activities in videos; an important visual cue that has long been undervalued. We revisit the conventional definition of activity and restrict it to Complex Action: a set of one-actions with a weak temporal pattern that serves a specific purpose. Related works use spatiotemporal 3D convolutions with fixed kernel size, too rigid to capture the varieties in temporal extents of complex actions, and too short for long-range temporal modeling. In contrast, we use multi-scale temporal convolutions, and we reduce the complexity of 3D convolutions. The outcome is Timeception convolution layers, which reasons about minute-long temporal patterns, a factor of 8 longer than best related works. As a result, Timeception achieves impressive accuracy in recognizing the human activities of Charades, Breakfast Actions and MultiTHUMOS. Further, we demonstrate that Timeception learns long-range temporal dependencies and tolerate temporal extents of complex actions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “, [100,110,139,221,243]) because in some cases a specific action can correspond to a specific state change.”
Action Genome: Actions As Compositions of Spatio-Temporal Scene Graphs (2019, methodology): “A major line of work has focused on developing powerful neural architectures to extract useful representations from videos [10, 23, 31, 69, 72].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “The authors of [45] revisited the definition of activity and restricted it to complex action which is a set of simple one-actions which compose the activity (e.”
Omni-sourced Webly-supervised Learning for Video Recognition (2020, methodology): “The former uses 2D networks to extract image-level feature and performs temporal aggregation [48,60,19] on top while the latter learns spatial-temporal features directly from video clips [45,9,44].”
Self-supervised Video Representation Learning by Pace Prediction (2020, background): “Video understanding, especially action recognition, has been extensively studied for decades, where video representation learning serves as the fundamental problem of other video-related tasks, such as complex action recognition [23], action temporal localization [8, 41, 42], video caption [49, 52],”
Ego-Topo: Environment Affordances From Egocentric Video (2020, methodology): “We would also like to thank Noureldien Hussein for their help with the Timeception [30] and Videograph [31] models.”
X3D: Expanding Architectures for Efficient Video Recognition (2020, unknown): “[2,5,9, 11, 13, 18, 28, 35, 38, 42, 44, 55, 56, 63, 65, 66, 72, 76, 84–86].”
Human Action Recognition From Various Data Modalities: A Review (2020, background): “[106] proposed multi-scale temporal-only convolutions, dubbed Timeception, to account for large variations and tolerate a variety of temporal extents in complex and long actions.”
A Multigrid Method for Efficiently Training Video Models (2019, background): “They are currently the state of the art for video understanding [3,9,12,19,30–32,34,43– 45, 47–49].”