Delving into Egocentric Actions

By Yin Li, Zhefan Ye, James M. Rehg. 2015

305 citations recorded by the source corpus.

We address the challenging problem of recognizing the camera wearer's actions from videos captured by an egocentric camera. Egocentric videos encode a rich set of signals regarding the camera wearer, including head movement, hand pose and gaze information. We propose to utilize these mid-level egocentric cues for egocentric action recognition. We present a novel set of egocentric features and show how they can be combined with motion and object features. The result is a compact representation with superior performance. In addition, we provide the first systematic evaluation of motion, object and egocentric cues in egocentric action recognition. Our benchmark leads to several surprising findings. These findings uncover the best practices for egocentric actions, with a significant performance boost over all previous state-of-the-art methods on three publicly available datasets.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Segment Anything (2023, background): “ADE20K[117] BBBC038v1[12] Cityscapes[25] DOORS[80] DRAM[24] EgoHOS[113] GTEA[34,63] Hypersim[86] IBD[17] iShape[111] LVIS[44] NDD20[100] NDISPark[22,23] OVIS[81] PPDLS[74] Plittersdorf[46] STREETS[91] TimberSeg[38] TrashCan[52] VISOR[28,27] WoodScape[112] PIDRay[104] ZeroWaste-f[6] Figure 8: Samples”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, background): “More recently, many works [1,73,4,34,36,37,58,59,61,24,50] have applied deep networks for hand or object segmentation.”
Rescaling Egocentric Vision: Collection, Pipeline and Challenges for EPIC-KITCHENS-100 (2020, methodology): “Table 4 also compares EPIC-KITCHENS-100 with other datasets used for action anticipation [30,54,55,56,61,62,40].”
Modeling Temporal Dynamics and Spatial Configurations of Actions Using Two-Stream Recurrent Neural Networks (2017, background): “Human action recognition [2] has become an active area in computer vision and there are many important research problems, such as event recognition [23], group based activities recognition [27], human object interactions [15] and activities in egocentric videos [29, 11].”
Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videos (2018, background): “For comparison, the largest concurrent egocentric dataset EGTEA Gaze+, has 28 hours (10,325 instances) [5].”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “Li et al. [39] studied how different egocentric cues, (including gaze, the presence of hands and objects, as well as head motion), can be used to perform the task.”
Contextual Attention for Hand Detection in the Wild (2019, background): “ColorHandPose [34] Specific poses 43,986 Synthetic HandNet [31] Only 10 subjects 212,928 Auto GTEA [20] Only 4 subjects 663 Manual Oxford-Hand [22] Unconstrained 2686 Manual”
Detecting Engagement in Egocentric Video (2016, background): “However, they do not address our scenario: 1) they learn models specific to the objects [34], [36]–[38], [40], [41] or scenes [39] with which the activity takes place (e.”
FaceShifter: Towards High Fidelity And Occlusion Aware Face Swapping (2019, methodology): “Occlusion images are randomly sampled from the EgoHands [30], GTEA Hand2K [31, 32, 33] and object renderings from ShapeNet [34].”
Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video (2017, background): “Comparative recognition results have been reported on these datasets in [29, 31, 27, 16, 22, 17].”
First-Person View Hand Segmentation of Multi-Modal Hand Activity Video Dataset (2020, methodology): “The traditional approach to create a RGB hand segmentation video dataset is through manual pixel-wise labeling [5, 7, 32, 50].”
Recognizing Micro-Actions and Reactions from Paired Egocentric Videos (2016, methodology): “In this way, egocentric videos are advantageous from a sensing perspective since the head motion and hand motion of camera wearers are often observed clearly in such videos, making it possible to perform various forms of first-person action recognition [7, 9, 13, 18, 25, 27, 30].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “unique challenges to action recognition due to fast camera motion, large occlusions and background clutter [33].”
What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019, background): “Action Recognition Our work is related to previous research on action recognition from third person vision [7, 13, 14, 30, 34, 35, 49, 57, 59, 60, 61, 67] and first person vision [11, 12, 36, 37, 38, 44, 48, 52, 54, 55, 56].”
First-Person Activity Forecasting with Online Inverse Reinforcement Learning (2016, background): “First-person vision (FPV): Wearable cameras have been used for various human behavior understanding tasks [4, 10, 12, 16, 22] because they give direct access to detailed visual information about a person’s actions.”
Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals (2017, methodology): “Hybrid approaches [43, 38, 56] utilize both object and motion information.”
See the Forest for the Trees: Joint Spatial and Temporal Recurrent Neural Networks for Video-Based Person Re-identification (2017, unknown): “Furthermore, videos contain richer information than a single image [19, 20], which is beneficial for identifying a person under complex conditions, including occlusions, and the changes of illumination and viewpoint.”
Leaving Some Stones Unturned: Dynamic Feature Prioritization for Activity Detection in Streaming Video (2016, background): “Some focus on recognition in a specific domain, such as egocentric video [23,24].”
SAM 2: Segment Anything in Images and Videos (2024, unknown): “…(Caicedo et al., 2019), DOORS (Pugliatti & Topputo, 2022), DRAM (Cohen et al., 2022), EgoHOS (Zhang et al., 2022), GTEA (Fathi et al., 2011; Li et al., 2015), iShape (Yang et al., 2021a), NDD20 (Trotter et al., 2020), NDISPark (Ciampi et al., 2021, 2022), OVIS (Qi et al., 2022), PPDLS…”