Pooled motion features for first-person videos

By Michael S. Ryoo, Brandon Rothrock, Larry H. Matthies. 2014

186 citations recorded by the source corpus.

In this paper, we present a new feature representation for first-person videos. In first-person video understanding (e.g., activity recognition), it is very important to capture both entire scene dynamics (i.e., egomotion) and salient local motion observed in videos. We describe a representation framework based on time series pooling, which is designed to ab] short-term/long-term changes in feature descriptor elements. The idea is to keep track of how descriptor values are changing over time and summarize them to represent motion in the activity video. The framework is general, handling any types of per-frame feature descriptors including conventional motion descriptors like histogram of optical flows (HOF) as well as appearance descriptors from more recent convolutional neural networks (CNN). We experimentally confirm that our approach clearly outperforms previous feature representations including bag-of-visual-words and improved Fisher vector (IFV) when using identical underlying feature descriptors. We also confirm that our feature representation has superior performance to existing state-of-the-art features like local spatio-temporal features and Improved Trajectory Features (originally developed for 3rd-person videos) when handling first-person videos. Multiple first-person activity datasets were tested under various settings to confirm these findings.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, neutral): “More recent approaches in egocentric perception leverage learned feature representations, utilizing pre-trained convolutional network features (Ryoo, Rothrock, and Matthies 2015), finetuning them (Ma, Fan, and Kitani 2016; Zellers et al. 2022), or training them from scratch (Bambach et al. 2015)…”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, background): “Ryoo et al. [40] proposed an approach to temporally pool features for egocentric action recognition.”
You2Me: Inferring Body Pose in Egocentric Video via First and Second Person Interactions (2019, background): “[14], activities [44, 16, 39, 56, 46, 37, 52, 51, 68], visible hand and arm pose [7, 31, 32, 49, 48], eye gaze [33], or anticipating future camera trajectories [41, 9].”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, methodology): “[90] developed a novel pooling method for deep models.”
In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video (2018, methodology): “[30] developed a novel pooling method for deep models.”
Trespassing the Boundaries: Labeling Temporal Bounds for Object Interactions in Egocentric Video (2017, background): “Comparative recognition results have been reported on these datasets in [29, 31, 27, 16, 22, 17].”
The Open World of Micro-Videos (2016, methodology): “…with the following simple pipeline: given a video, (1) run oﬀ-the-shelf VGG-16 models [33] on 15 equally-spaced frames from a video, (2) for each frame, extract (6144-dimensional) multi-scale features across multiple layers [39], and (4) max-pool the resulting features across the 15 frames [36].”
H+O: Unified Egocentric Recognition of 3D Hand-Object Poses and Interactions (2019, background): “Early approaches for egocentric action recognition rely on motion cues [29, 50, 55].”
UvA-DARE (Digital Academic Repository) Dynamic Image Networks for Action Recognition (2016, methodology): “Recent works such as [5, 6, 7, 9, 22] pointed out that long term dynamics and temporal patterns are a very important cues for the recognition of actions.”
Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals (2017, methodology): “Motion-based approaches [33, 51, 55, 68, 57] assume that each activity creates coherent motion patterns.”
First Person Action-Object Detection with EgoNet (2016, background): “However, due to different visual statistics between natural images and first person images and a limited number of annotated firstperson data, first person vision has not been fully exploited by deep learning techniques, except with the pre-trained models as generic feature extractors [20, 23, 45].”
Going Deeper into First-Person Activity Recognition (2016, methodology): “[29] develops a new pooled feature representation and shows superior performance using CNN as a appearance feature extractor.”
Recognizing Micro-Actions and Reactions from Paired Egocentric Videos (2016, methodology): “In PoTCD [27, 30], we used only three pyramids (split into one, two, and four segments) as we worked on short video clips.”
What Would You Expect? Anticipating Egocentric Actions With Rolling-Unrolling LSTMs and Modality Attention (2019, background): “Action Recognition Our work is related to previous research on action recognition from third person vision [7, 13, 14, 30, 34, 35, 49, 57, 59, 60, 61, 67] and first person vision [11, 12, 36, 37, 38, 44, 48, 52, 54, 55, 56].”