Going Deeper into First-Person Activity Recognition

By Minghuang Ma, Haoqi Fan, Kris M. Kitani. 2016

317 citations recorded by the source corpus.

We bring together ideas from recent work on feature design for egocentric action recognition under one framework by exploring the use of deep convolutional neural networks (CNN). Recent work has shown that features such as hand appearance, object attributes, local hand motion and camera ego-motion are important for characterizing first-person actions. To integrate these ideas under one framework, we propose a twin stream network architecture, where one stream analyzes appearance information and the other stream analyzes motion information. Our appearance stream encodes prior knowledge of the egocentric paradigm by explicitly training the network to segment hands and localize objects. By visualizing certain neuron activation of our network, we show that our proposed architecture naturally learns features that capture object attributes and hand-object configurations. Our extensive experiments on benchmark egocentric action datasets show that our deep architecture enables recognition rates that significantly outperform state-of-the-art techniques - an average 6:6% increase in accuracy over all datasets. Furthermore, by learning to recognize objects, actions and activities jointly, the performance of individual recognition tasks also increase by 30% (actions) and 14% (objects). We also include the results of extensive ablative analysis to highlight the importance of network design decisions.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, neutral): “…in egocentric perception leverage learned feature representations, utilizing pre-trained convolutional network features (Ryoo, Rothrock, and Matthies 2015), finetuning them (Ma, Fan, and Kitani 2016; Zellers et al. 2022), or training them from scratch (Bambach et al. 2015) with first-person videos.”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “Ma et al. [41] designed a deep learning architecture which allows to integrate different egocentric-based features to recognize actions.”
Cross-Domain First Person Audio-Visual Action Recognition through Relative Norm Alignment (2021, background): “Since egocentric action recognition suffers from the motion of the camera and sudden changes of view, the main approaches proposed so far are based on multistream architectures [7, 46, 39, 34, 8, 27, 38], many of which are inherited from the third-person action recognition literature.”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, background): “Egocentric video analysis Recent egocentric vision work focuses primarily on recognizing objects [13], activities [38, 15, 33, 48, 40, 32, 44, 43, 58], visible hand and arm poses [7, 28, 29, 42], eye gaze [30], or anticipating future camera trajectories [35, 9].”
Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition (2019, background): “to study this fundamental problem [5], [6], [12], [43], [44], [80]–[82].”
Future Person Localization in First-Person Videos (2017, background): “Recently, some work has focused on activity recognition [7, 22, 23, 28], activity forecasting [6, 9, 26, 31], person identification [11], gaze anticipation [45] and grasp recognition [3, 4, 21, 35].”
Ego-Topo: Environment Affordances From Egocentric Video (2020, background): “Recent work to recognize or anticipate actions in egocentric video adopts state-of-the-art video models from third-person video, like two-stream networks [42, 47], 3DConv models [6, 54, 49], or recurrent networks [15, 16, 62, 66].”
Is First Person Vision Challenging for Object Tracking? (2020, unknown): “On a more abstract level, the features computed after the frame by frame localization of objects have been increasingly used for egocentric action recognition [81, 83, 56] and anticipation [30, 72].”
Next-active-object prediction from egocentric videos (2017, background): “Indeed, as observed in [17,18,23], it is advantageous to decompose long term egocentric activities in terms of “atomic actions” and interactions with objects to improve the final activity recognition task.”
Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation (2018, background): “[21] designed a CNN architecture to integrate different egocentric cues for the recognition of egocentric actions and activities.”
Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video (2016, background): “Some focus on recognition in a specific domain, such as egocentric video [23,24].”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, methodology): “We were unable to compare against relevant methods from [3], [19].”
The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain (2020, background): “While past works have already investigated the tasks of action recognition [13, 38, 42, 57], active object detection [47], and active object recognition [13] in the context of egocentric vision, Human-Object Interaction (HOI) detection has been generally studied in the context of third person vision”
Egocentric Hand Track and Object-Based Human Action Recognition (2019, background): “Another multimodal approach for egocentric activity recognition is described in [19].”
Are They Going to Cross? A Benchmark Dataset and Baseline for Pedestrian Crosswalk Behavior (2017, background): “In the computer vision community human activity recognition has been extensively studied [26, 23, 17, 6, 41].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “In addition to motion, features based on gaze information [13], head motion [33], and, recently, CNN-learned features [35] have been proposed by a number of studies.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “Egocentric cues in video understanding models Egocentric video offers several unique cues that have been leveraged to improve video understanding models [39, 46].”
First Person Action-Object Detection with EgoNet (2016, background): “Some prior work focused specifically on handled object detection [44, 6].”
People with Visual Impairment Training Personal Object Recognizers: Feasibility and Challenges (2017, methodology): “Based on our observation of the users tendency to use their hand as a guiding point of reference for the camera, we will incorporate elements from our prior work on firstperson activity recognition [11] to select training examples where the object appears in the vicinity of the user’s hand.”
What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019, methodology): “Specifically, we build on previous ideas investigated in the context of action recognition such as the use of multiple modalities for video analysis [49], the use of Temporal Segment Networks [61] as a principled way to train CNNs for action recognition, as well as the explicit encoding of object-ba”