What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention

By Antonino Furnari, Giovanni Maria Farinella. 2019

206 citations recorded by the source corpus.

Egocentric action anticipation consists in understanding which objects the camera wearer will interact with in the near future and which actions they will perform. We tackle the problem proposing an architecture able to anticipate actions at multiple temporal scales using two LSTMs to 1) summarize the past, and 2) formulate predictions about the future. The input video is processed considering three complimentary modalities: appearance (RGB), motion (optical flow) and objects (object-based features). Modality-specific predictions are fused using a novel Modality ATTention (MATT) mechanism which learns to weigh modalities in an adaptive fashion. Extensive evaluations on two large-scale benchmark datasets show that our method outperforms prior art by up to +7% on the challenging EPIC-Kitchens dataset including more than 2500 actions, and generalizes to EGTEA Gaze+. Our approach is also shown to generalize to the tasks of early action recognition and action recognition. Our method is ranked first in the public leaderboard of the EPIC-Kitchens egocentric action anticipation challenge 2019. Please see the project web page for code and additional details: http://iplab.dmi.unict.it/rulstm.

Subfields: Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “, video + audio), and (Furnari and Farinella 2019) studies future action anticipation from egocentric videos.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “[76] Antonino Furnari and Giovanni Maria Farinella.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “We use our prior work [81] as a baseline.”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “What space should the model predict on? Recent approaches [25,26,28,72] have been proposed to predict the discrete future action category given a sequence of frames as inputs, namely action anticipation.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “Model architectures proposed for these tasks include multi-stream networks [46, 38, 35, 68], recurrent networks [22, 21, 65], 3D conv nets [53, 44] and spatially grounded topological graph models [50].”
Temporal Aggregate Representations for Long-Range Video Understanding (2020, methodology): “We experiment on Breakfast [18], 50Salads [34] and EPIC-Kitchens [5].”
Supplementary Materials for: “Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video” (2020, unknown): “We highlight our comparison to RULSTM.”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “Similarly, in action anticipation and human motion forecasting, high-level semantic or low level actions are predicted using visual history [1, 11,19,22,31–33,37,40,46–48,55,58,72,75,100,119,120].”
Procedure Planning in Instructional Videos (2019, background): “The others ask the question “What is going to happen?”, such as early action recognition [9, 35], action label prediction [1, 6, 9, 14, 22, 27, 29, 36], video prediction [18–20, 24, 26, 34], etc.”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, methodology): “We computed objectcentric features following [33], while, gaze features have been obtained weighting the object-centric features with the distance between the center of objects bounding boxes and the gaze position in the image.”
Temporal Pyramid Network for Action Recognition (2020, methodology): “Following [8], we randomly select 232 videos (23439 segments) for training and 40 videos (4979 segments) for validation.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, methodology): “Several papers used simple bag-of-words representations of objects present in the frame [10, 49, 53], while others encoded objects in bounding-box representations [109], or even”
Learning Temporal Dynamics from Cycles in Narrated Video (2021, background): “Recent work extends this, using contrastive learning or other techniques to predict future representations [13, 15, 16, 47, 60].”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, background): “Ourdataset is publicly available(1) andhas been considered by several recentworks [22], [23], [24], [25], [26], [27].”
Anticipative Video Transformer (2021, methodology): “First, we compare to the state-of-theart RULSTM method using only our AVT (head) model applied to the exact same features from TSN [88] trained for classification on EK100.”
VLN↻BERT: A Recurrent Vision-and-Language BERT for Navigation (2020, background): “However, we believe that it has a huge potential in addressing other tasks which require sequential interactions/decisions, such as language and visual dialog [5, 8, 21, 25, 45], dialog navigation [9, 40, 41] and action anticipation for reactive robot response [1, 13, 20].”
Is First Person Vision Challenging for Object Tracking? (2020, unknown): “Since many FPV tasks such as object interaction [20] and early action recognition [34], or action anticipation [19], require real-time computation, we evaluated the ability of trackers to provide their object localization in such a setting.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “…to encourage efﬁcient RL. Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [69, 61, 8, 19], including analyzing hand-object interactions [55, 2, 60, 7, 24] and egocentric video understanding [34, 21, 37, 31, 13].”