When will you do what? - Anticipating Temporal Occurrences of Activities

By Yazan Abu Farha, Alexander Richard, Juergen Gall. 2018

219 citations recorded by the source corpus.

Analyzing human actions in videos has gained increased attention recently. While most works focus on classifying and labeling observed video frames or anticipating the very recent future, making long-term predictions over more than just a few seconds is a task with many practical applications that has not yet been addressed. In this paper, we propose two methods to predict a considerably large amount of future actions and their durations. Both, a CNN and an RNN are trained to learn future video labels based on previously seen content. We show that our methods generate accurate predictions of the future even for long videos with a huge amount of different actions and can even deal with noisy or erroneous input information.

Subfields: Computer Vision and Pattern Recognition, Time-Series & Forecasting

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Time-Series & Forecasting

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “…video offers a host of interesting challenges, such as human-object interactions [26,46,163], activity recognition [110,139,243], anticipation [4,75,86,144,205], video summarization [48, 129,131,147,148,232], detecting hands [16,134], parsing social interactions [66,168,231], and inferring…”
Contrastive Bidirectional Transformer for Temporal Representation Learning (2019, methodology): “For the Breakfast dataset where we have more training examples the fully-supervised approach [1] performs on par with our method.”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “Abu et al. [12] explored the use of CNNs and RNNs to predict the occurrence of future actions based on past observations.”
Ego-Topo: Environment Affordances From Egocentric Video (2020, methodology): “Note that state-of-the-art anticipation models [15, 1, 35] —which decode future actions from such an encoding of past (observed) video—address an orthogonal problem, and can be used in parallel with our method.”
Procedure Planning in Instructional Videos (2019, methodology): “This ablation can also be seen as a re-implementation of the RNN-based model in [1].”
Zero-Shot Anticipation for Instructional Activities (2018, methodology): “Recently, [5] predicts multiple actions into the future; our method also predicts multiple steps but unlike [5], we do not require repetitions of activity sequences for training.”
Anticipative Video Transformer (2021, methodology): “We observe it outperforms previous RNN [2] and attention [74] based approaches by a significant 7.”
Egocentric Video-Language Pretraining (2022, background): “Egocentric videos, collected by participants using wearable cameras, offer a natural perspective of people’s daily activities and raise a range of challenging research topics [11, 12, 30].”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “It is the subject of a recent surge in research interest in first-person activity recognition [35, 77], anticipation [21, 1], and video summarization [36, 74, 13] with many valuable future applications in augmented reality and robotics.”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, methodology): “Previous methods [1,15] to date require having already segmented temporal observations.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, methodology): “As an example, it is common to ﬁnd works using the term “action anticipation” to refer to the tasks of predicting an on-going action [6, 7], an action in the near future [8, 9, 10], or an action in the distant future [11, 12].”
Learning Video Representations using Contrastive Bidirectional Transformer (2019, methodology): ““mix ingredients”. Annotations are also deﬁned at the level of action units by start and end time. Videos have on average a length of 6.4 minutes and contain on average 20 action instances. Following [23, 1], we use the ﬁve-fold cross-validation resulting in an average of 722 training and 180 test e”
Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation (2018, methodology): “[1] designed two methods to predict future actions and their durations.”
What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019, methodology): “Previous works investigated different forms of action and activity anticipation from third person video [1, 15, 18, 25, 27, 28, 31, 32, 33, 40, 58, 64].”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Similarly, in action anticipation and human motion forecasting, high-level semantic or low level actions are predicted using visual history [1, 11,19,22,31–33,37,40,46–48,55,58,72,75,100,119,120].”
Global Regularizer and Temporal-Aware Cross-Entropy for Skeleton-Based Early Action Recognition (2018, methodology): “[33] introduced two different architectures based on CNN and RNN for future action anticipation.”
EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought (2023, background): “Egocentric videos, which are captured using wearable cameras, provide a natural perspective of daily activities and pose several challenging research questions [25, 26, 27].”