Temporal segmentation and activity classification from first-person sensing

By Ekaterina H. Spriggs, Fernando De la Torre, Martial Hebert. 2009

293 citations recorded by the source corpus.

Temporal segmentation of human motion into actions is central to the understanding and building of computational models of human motion and activity recognition. Several issues contribute to the challenge of temporal segmentation and classification of human motion. These include the large variability in the temporal scale and periodicity of human actions, the complexity of representing articulated motion, and the exponential nature of all possible movement combinations. We provide initial results from investigating two distinct problems -classification of the overall task being performed, and the more difficult problem of classifying individual frames over time into specific actions. We explore first-person sensing through a wearable camera and inertial measurement units (IMUs) for temporally segmenting human motion into actions and performing activity classification in the context of cooking and recipe preparation in a natural environment. We present baseline results for supervised and unsupervised temporal segmentation, and recipe recognition in the CMU-multimodal activity database (CMU-MMAC).

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

In Computer Vision and Pattern Recognition

In Control and Systems Engineering

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “, optical flows) [98, 99] as well as features from human gaze, hands, and objects [100, 101, 102, 103, 104, 105].”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “Spriggs et al. [35] investigated the problem of supervised and unsupervised action segmentation using Inertial Measurement Units (IMU) and egocentric video.”
A database for fine grained activity detection of cooking activities (2012, background): “Another direction is to detect activities with a body-worn camera [34].”
Understanding egocentric activities (2011, background): “[30] segment and classify daily activities from a first-person view using accelerometers and visual information.”
Predicting Important Objects for Egocentric Video Summarization (2015, background): “Recent methods explore activity recognition [34,35,12,36], handled object recognition [37], novelty detection [38], hand detection [39], gaze prediction [40], social interaction analysis [41], or activity discovery for nonvisual sensory data [42].”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, background): “[14], activities [44, 16, 39, 56, 46, 37, 52, 51, 68], visible hand and arm pose [7, 31, 32, 49, 48], eye gaze [33], or anticipating future camera trajectories [41, 9].”
An end-to-end generative framework for video segmentation and recognition (2015, methodology): “Figure 4: Sample frames from the datasets used for performance evaluation: a) ADL [14], b) Olympic [16], c) ToyAssembly [32], d) CMU-MMAC [29], e) MPIICooking [23], f) 50Salads [30], g) Breakfast [12], and h) CRIM13 [2].”
A Survey on Activity Detection and Classification Using Wearable Sensors (2017, background): “[214] focus on temporal segmentation of an activity in order to recognize different temporal parts.”
Fast unsupervised ego-action learning for first-person sports videos (2011, background): “working at the office, eating dinner) [4] and segmenting a signal into primitive units [9].”
Leveraging Uncertainty to Rethink Loss Functions and Evaluation Measures for Egocentric Action Anticipation (2018, methodology): “[30] employed Inertial Measurement Units (IMU) and a wearable camera to segment an egocentric video into action segments.”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, background): “[86] proposed to segment and recognize daily activities using a combination of video and wearable sensor data.”
A review on vision techniques applied to Human Behaviour Analysis for Ambient-Assisted Living (2012, methodology): “Other sensing devices are also used in diverse works [2, 8, 63, 109, 132], or a combination of vision and RFID tags [129] either as a direct way to recognise the objects being manipulated, or as a means for supervised learning of visual object appearance (using the correspondence between the RFID an”
BABEL: Bodies, Action and Behavior with English Labels (2021, unknown): “The CMU MMAC dataset [32] contains precise frame labels for a ﬁxed set of 17 cooking actions (including ‘none’).”