Encouraging LSTMs to Anticipate Actions Very Early

By Mohammad Sadegh Ali Akbarian, F. Saleh, Mathieu Salzmann, Basura Fernando, Lars Petersson, Lars Andersson. 2017

181 citations recorded by the source corpus.

In contrast to the widely studied problem of recognizing an action given a complete sequence, action anticipation aims to identify the action from only partially available videos. As such, it is therefore key to the success of computer vision applications requiring to react as early as possible, such as autonomous navigation. In this paper, we propose a new action anticipation method that achieves high prediction accuracy even in the presence of a very small percentage of a video sequence. To this end, we develop a multi-stage LSTM architecture that leverages context-aware and action-aware features, and introduce a novel loss function that encourages the model to predict the correct class as early as possible. Our experiments on standard benchmark datasets evidence the benefits of our approach; We outperform the state-of-the-art action anticipation methods for early prediction by a relative increase in accuracy of 22.0% on JHMDB-21, 14.0% on UT-Interaction and 49.9% on UCF-101.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

RGB-D-based Human Motion Recognition with Deep Learning: A Survey (2017, background): “Predicting future actions before they are actually executed is a critical ingredient for enabling us to effectively interact with other humans on a daily basis [104, 51, 69, 137, 105].”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, background): “Aliakbarian et al. [5] proposed a two stage LSTM architecture which models context and action to perform early action recognition.”
Global Regularizer and Temporal-Aware Cross-Entropy for Skeleton-Based Early Action Recognition (2018, methodology): “Compared to the method in [32] and [31], the improvements of the proposed EARN + GR + TCE are by 1.”
Future Event Prediction: If and When (2019, methodology): “[3, 2] use context- and action-aware features with a two-stage LSTM to recognize actions from partial video sequences.”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, background): “Rather than predicting pixels, another line of work uses supervision to predict future action labels [19, 24, 28, 36, 45].”
When will you do what? - Anticipating Temporal Occurrences of Activities (2018, methodology): “While early activity detection addresses the problem of inferring the class label of an action at the point when the activity starts or shortly thereafter [23, 5, 14, 24], other works predict the class label of the action that will happen next [19, 6, 9].”
Peeking Into the Future: Predicting Future Person Activities and Locations in Videos (2019, background): “[20] and [2] proposed different losses to encourage LSTM to recognize actions early in internet videos.”
What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019, methodology): “Additionally, tasks such as action anticipation [18, 32, 58] and early action recognition [2, 9, 39] pose a series of key challenges from a computational perspective.”
Progressive Teacher-Student Learning for Early Action Prediction (2019, background): “Different from action recognition, where actions are assumed to be fully executed and observed, early action prediction aims to recognize actions before they are completely executed [32, 25, 23, 24, 2, 16, 22].”
Skip RNN: Learning to Skip State Updates in Recurrent Neural Networks (2017, background): “the cost can be increased at each time step to encourage the network to emit a decision earlier [1], or the number of updates can be strictly bounded and enforced.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, methodology): “As an example, it is common to ﬁnd works using the term “action anticipation” to refer to the tasks of predicting an on-going action [6, 7], an action in the near future [8, 9, 10], or an action in the distant future [11, 12].”
Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation (2018, background): “[2] design a new loss function for early action recognition which softens the penalization of false positives when the action has been only partially observed.”