Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video

By Antonino Furnari, Giovanni Maria Farinella. 2020

177 citations recorded by the source corpus.

In this paper, we tackle the problem of egocentric action anticipation, i.e., predicting what actions the camera wearer will perform in the near future and which objects they will interact with. Specifically, we contribute Rolling-Unrolling LSTM, a learning architecture to anticipate actions from egocentric videos. The method is based on three components: 1) an architecture comprised of two LSTMs to model the sub-tasks of summarizing the past and inferring the future, 2) a Sequence Completion Pre-Training technique which encourages the LSTMs to focus on the different sub-tasks, and 3) a Modality ATTention (MATT) mechanism to efficiently fuse multi-modal predictions performed by processing RGB frames, optical flow fields and object-based features. The proposed approach is validated on EPIC-Kitchens, EGTEA Gaze+ and ActivityNet. The experiments show that the proposed architecture is state-of-the-art in the domain of egocentric videos, achieving top performances in the 2019 EPIC-Kitchens egocentric action anticipation challenge. The approach also achieves competitive performance on ActivityNet with respect to methods not based on unsupervised pre-training and generalizes to the tasks of early action recognition and action recognition. To encourage research on this challenging topic, we made our code, trained models, and pre-extracted features available at our web page: http://iplab.dmi.unict.it/rulstm.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

MERLOT RESERVE: Neural Script Knowledge through Vision and Language and Sound (2022, background): “All prior work reported on the official leaderboard use supervision from the in-domain training set, which we do not use at all [45, 38].”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[75] Antonino Furnari and Giovanni Farinella.”
Look Before you Speak: Visually Contextualized Utterances (2020, methodology): “In addition to the baselines described in Section 5.2, we also compare to two state-of-the-art vision-only models for next step prediction, RULSTM [24] and TAR [66]6 Pretraining on BERT and using the MLM loss particularly help prevent our multimodal model from overfitting on this small dataset.”
Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, methodology): “The first exploits Long Short-Term Memory and variants [50, 51, 49, 18] to generate an embedding representation based on the temporal relations between the features frames.”
Is First Person Vision Challenging for Object Tracking? (2020, unknown): “to extract features) and movements in the scene [81, 29, 30, 72].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, methodology): “In this work, we adopt the RULSTM model [34] to evaluate state-of-the-art action anticipation methods on the proposed MECCANO dataset also considering the gaze and depth signals.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “It is the subject of a recent surge in research interest in first-person activity recognition [35, 77], anticipation [21, 1], and video summarization [36, 74, 13] with many valuable future applications in augmented reality and robotics.”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “What space should the model predict on? Recent approaches [25,26,28,72] have been proposed to predict the discrete future action category given a sequence of frames as inputs, namely action anticipation.”
Anticipative Video Transformer (2021, background): “First, we compare to the state-of-theart RULSTM method using only our AVT (head) model applied to the exact same features from TSN [88] trained for classification on EK100.”