In the Eye of Beholder: Joint Learning of Gaze and Actions in First Person Video

By Yin Li, Miao Liu, James M. Rehg. 2018

395 citations recorded by the source corpus.

Subfields: Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language (2022, background): “Similar to the topics explored in early work, learning of visual representations capturing human hands, objects, and eye gaze has been extensively studied (Garcia-Hernando et al. 2018; Li, Liu, and Rehg 2018).”
Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications (2022, methodology): “The other two datasets EYTH [73] and EGTEA [33] provide only binary mask labels for both hands without differentiating between left or right.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, methodology): “Others are (semi-)scripted, where camera wearers are instructed to perform a certain activity, as in Charades-Ego [201] and EGTEA [138].”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, methodology): “Existing works have proposed to use a learned gaze prediction model to guide the process of learning a human policy [Li et al. , 2018; Zhang et al. , 2018; Kim et al. , 2018].”
Rolling-Unrolling LSTMs for Action Anticipation from First-Person Video (2020, methodology): “Li et al. [16] introduced a deep learning model for joint gaze estimation and action recognition in egocentric video.”
Multi-Modal Domain Adaptation for Fine-Grained Action Recognition (2019, background): “Figure 2: Fine-grained action datasets [8, 17, 26, 28, 38, 42, 46, 47, 50], x-axis: number of action segments per environment (ape), y-axis: dataset size divided by ape.”
Review of Eye Tracking Metrics Involved in Emotional and Cognitive Processes (2021, methodology): “around the eye, such as eyebrows or eyelashes [69].”
EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset (2020, background): “While recent datasets like EPIC-KITCHENS (Damen et al., 2018) and GTEA Gaze+ (Li et al., 2018) have signiﬁcantly advanced this goal, there is no public egocentric dataset addressing two key elements of embodied intelligence: natural language and multi-perspective interaction.”
LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activities (2020, background): “0 RGB 2017 EGTEA GAZE+ [34] 3 7 7 7 86 2.”
Human Gaze Assisted Artificial Intelligence: A Review (2020, methodology): “Convolutional neural networks remain the most popular tool for predicting human attention [Li et al. , 2018; Zhang et al. , 2018a; Palazzi et al. , 2018; Deng et al. , 2019].”
Grouped Spatial-Temporal Aggregation for Efficient Action Recognition (2019, methodology): “For the EGTEA Gaze++ dataset, it contains 106 classes with 19 different verbs.”
Joint Hand Motion and Interaction Hotspots Prediction from Egocentric Videos (2022, methodology): “Training takes 35 epochs on Epic-Kitchens, 25 epochs on EGTEA Gaze+, including 5 epochs warm-up [31] and rest epochs with cosine decay [52].”
H2O: Two Hands Manipulating Objects for First Person Interaction Recognition (2021, background): “While many datasets for thirdperson action recognition have been proposed throughout the years [30, 40, 69, 96], recently there is a surge in interest for data targeting also egocentric scenarios [12, 13, 29, 47, 60, 68] that involve mostly 2D features.”
Ego-Exo: Transferring Visual Representations from Third-person to First-person Videos (2021, background): “Prior work shows it is possible to recognize a camera wearer’s actions by attending to only a small region around the gaze point [38], as gaze often focuses on hand-object manipulation.”
Supplementary Materials for: “Forecasting Human-Object Interaction: Joint Prediction of Motor Attention and Actions in First Person Video” (2020, positive): “We followed [3] to use the features from bottom layers of the network for motor attention prediction and interaction hotspots estimation, and the features from the top layer for action anticipation.”
Shaping embodied agent behavior with activity-context priors from egocentric video (2021, background): “…to encourage efﬁcient RL. Learning about human actions from video Substantial work in computer vision explores models for human action recognition in video [69, 61, 8, 19], including analyzing hand-object interactions [55, 2, 60, 7, 24] and egocentric video understanding [34, 21, 37, 31, 13].”
Ego2Hands: A Dataset for Egocentric Two-hand Segmentation and Detection (2020, background): “Note that in the case of inter-hand occlusion and cross-over (e.g. t = 99 in row 1), the model trained on EGTEA mistakenly predicts the right hand as the left hand.”
WorkingHands: A Hand-Tool Assembly Dataset for Image Segmentation and Activity Mining (2019, methodology): “InterSegHands [5] 0 52K Y †† 28 EGTEA Gaze+ [18] 14K N Man.”