Next-active-object prediction from egocentric videos

By Antonino Furnari, Sebastiano Battiato, Kristen Grauman, Giovanni Maria Farinella. 2017

110 citations recorded by the source corpus.

Abstract Although First Person Vision systems can sense the environment from the user’s perspective, they are generally unable to predict his intentions and goals. Since human activities can be decomposed in terms of atomic actions and interactions with objects, intelligent wearable systems would benefit from the ability to anticipate user-object interactions. Even if this task is not trivial, the First Person Vision paradigm can provide important cues to address this challenge. We propose to exploit the dynamics of the scene to recognize next-active-objects before an object interaction begins. We train a classifier to discriminate trajectories leading to an object activation from all others and forecast next-active-objects by analyzing fixed-length trajectory segments within a temporal sliding window. The proposed method compares favorably with respect to several baselines on the Activity of Daily Living (ADL) egocentric dataset comprising 10 h of videos acquired by 20 subjects while performing unconstrained interactions with several objects.

Subfields: Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Previous work considers future localization [113, 120, 174, 230], action anticipation [76, 77, 86, 118, 127, 219], next active object prediction [20, 74], future event prediction [149,167], and future frame prediction [145,146,153,215,218,227].”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, background): “predicting future localization [49] or next-active object [50].”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “[61] proposed to anticipate human-object interactions by analyzing the motion of objects in egocentric video.”
Predicting the Future from First Person (Egocentric) Vision: A Survey (2021, methodology): “Predicting next active objects consists in detecting the objects in the scene which will be used by the camera wearer as part of a human-object interaction [59].”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, background): “Previous works have investigated different anticipation tasks from an egocentric perspective, including predicting future localisation [62], next-active objects [63] or forecasting actions [64].”
Affordances from Human Videos as a Versatile Representation for Robotics (2023, background): “More general interaction information can be learned from large human datasets [18–20, 39, 59, 63], to predict object information [29, 129] (RGB & 3D) [10], graphs [23] or environment information [27, 76] such as heatmaps [38, 75].”
MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain (2022, background): “The authors of [32] have been the ﬁrst to explore the next-active objects prediction problem.”
Grounded Human-Object Interaction Hotspots From Video (2018, methodology): “Future prediction has been applied to action anticipation [26, 55, 32, 44], active-object forecasting [16], and to guide demonstration learning in robotics [13, 14, 11].”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, background): “In general, there are two main categories of prediction, next-active object [5,18,24] and interaction hotspots [49, 54, 61].”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “More closely related to our work, visual affordance models derived from video can detect likely places for actions to occur [38, 18, 40], such as where to grasp a frying pan, and object saliency models can identify human-useable objects in egocentric video [14, 5, 20].”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “Recent action anticipation [13, 75, 15, 5, 53, 16, 61] predicts the immediate next action (e.”
Egocentric Hand Track and Object-Based Human Action Recognition (2019, methodology): “In [14], the trajectories of detected objects are computed and then classified as active or passive based on hand manipulations towards them.”
EgoVQA - An Egocentric Video Question Answering Benchmark Dataset (2019, background): “Recent computer vision research topics on studying first-person videos have included object detection [19,24,25,39], hands detection [10], scene understanding [26], activity recognition [23, 47], person identification and segmentation [20, 40, 57, 58], gaze detection and tracking [34, 62], etc.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “These include attention mechanisms from gaze and motor attention [47, 38, 42], active object detection [20, 7, 15, 70], and hands in contact [66, 60, 33].”
Is First Person Vision Challenging for Object Tracking? (2020, unknown): “to extract features) and movements in the scene [81, 29, 30, 72].”
Future Person Localization in First-Person Videos (2017, background): “Recently, some work has focused on activity recognition [7, 22, 23, 28], activity forecasting [6, 9, 26, 31], person identification [11], gaze anticipation [45] and grasp recognition [3, 4, 21, 35].”
What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019, methodology): “While a direct comparison of the proposed approach with the aforementioned works is unfeasible due to the lack of a common framework, our method incorporates some ideas from past approaches, such as the analysis of past actions [47] and the detection of the objects present in the scene to infer futu”