Social interactions: A first-person perspective

By Alireza Fathi, Jessica K. Hodgins, James M. Rehg. 2012

339 citations recorded by the source corpus.

This paper presents a method for the detection and recognition of social interactions in a day-long first-person video of u social event, like a trip to an amusement park. The location and orientation of faces are estimated and used to compute the line of sight for each face. The context provided by all the faces in a frame is used to convert the lines of sight into locations in space to which individuals attend. Further, individuals are assigned roles based on their patterns of attention. The rotes and locations of individuals are analyzed over time to detect and recognize the types of social interactions. In addition to patterns of face locations and attention, the head movements of the first-person can provide additional useful cues as to their attentional focus. We demonstrate encouraging results on detection and recognition of social interactions in first-person videos captured from multiple days of experience in amusement parks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Cognitive Science & Psychology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Cognitive Science & Psychology

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “The practice of giving cameras to participants to take out of the lab, first explored in [66,129,179], inspires our approach.”
Scaling Egocentric Vision: The EPIC-KITCHENS Dataset (2018, background): “We exclude egocentric datasets that focus on inter-person interactions [2, 12, 40], as these target a different research question.”
Multi-Stream Deep Neural Networks for RGB-D Egocentric Action Recognition (2019, neutral): “Disney World [40] 18 113 RGB Head – 6 2012 ! #”
The EPIC-KITCHENS Dataset: Collection, Challenges and Baselines (2020, background): “We exclude egocentric datasets that focus on inter-person interactions [29], [30], [31] as well as instructional videos [32], [33], [34], [35], as”
Summarization of Egocentric Videos: A Comprehensive Survey (2017, methodology): “4) Disneyworld [41]: This dataset was originally recorded to evaluate social interactions during a full day at an amusement park.”
EgoCom: A Multi-Person Multi-Modal Egocentric Communications Dataset (2020, background): “EgoCom differs from (Fathi et al., 2012) and (Yonetani et al., 2016) in that it captures multi-perspective multi-modal communication signals that can be exploited beyond vision tasks.”
Illusory Motion Reproduced by Deep Neural Networks Trained for Prediction (2018, methodology): “Models were trained with mean-squared error using videos from the First-Person Social Interactions Dataset (Fathi et al., 2012”
Predicting Important Objects for Egocentric Video Summarization (2015, background): “Recent methods explore activity recognition [34,35,12,36], handled object recognition [37], novelty detection [38], hand detection [39], gaze prediction [40], social interaction analysis [41], or activity discovery for non-visual sensory data [42].”
Detecting Engagement in Egocentric Video (2016, methodology): “Recent methods explore ways to predict the “importance” of spatial regions (objects, people) using cues like hand detection and frame centrality [28], [29], detect novelty [30], and infer “social saliency” when multiple cameras capture the same event [31]–[33].”
Visual Motif Discovery via First-Person Vision (2016, methodology): “, not only pausing but using hands [4, 5, 8, 9] and conversing with others [40], which are all informative for recognizing a variety of visual motifs.”
Egocentric Video-Language Pretraining (2022, neutral): “5 M HowTo100M [10] instructional 134 K 136 M 136 M 3rd-person view Charades-Ego [17] home 34 30 K 30 K UT-Ego [18] diverse 37 11 K 11 K Disneyworld [19] disneyland 42 15 K 15 K EPIC-KITCHENS-100 [20] kitchen 100 90 K 90 K EgoClip diverse 2 .”
A Review of Human Activity Recognition Methods (2015, methodology): “are weighted based on how often a person’s face is captured by a subject’s camera (Fathi et al., 2012).”
Jointly Learning Energy Expenditures and Activities Using Egocentric Multimodal Signals (2017, background): “Adding mid-level cues such as face, gaze, and hands has also been investigated by [37, 19, 18, 49, 6].”
Multitype Activity Recognition in Robot-Centric Scenarios (2015, background): “Detecting who is interacting with whom has been recently studied by Fathi et al. [19]: given a set of persons, their goal is to classify monologues, dialogues, and discussions.”
Privacy-Preserving Visual Learning Using Doubly Permuted Homomorphic Encryption (2017, background): “By allowing the cloud administrator to curate these videos, he will be able to learn activity classifiers [19, 20, 37, 45, 48, 52] from realistic and diverse data.”
Action recognition in RGB-D egocentric videos (2017, background): “For these reasons, increasing works have been proposed to investigate the egocentric videos from different aspects, like video summarization [15–17], visual recognition [18–21], social interaction [22], gaze detection [23] and many others.”
In the Eye of the Beholder: Gaze and Actions in First Person Video (2020, neutral): “Other efforts included the modeling of conversations [88] and reactions [8], [89] in social”
Detecting People Looking at Each Other in Videos (2013, background): “…(Laptev et al. 2008; Liu et al. 2009; Marín-Jiménez and Pérez de la Blanca 2012; Raptis et al. 2012; Sadanand and Corso 2012), and social (Fathi et al. 2012) and speciﬁc interaction recognition (e.g. hugging, Putting interactions together with previous character identiﬁ-cation work, it…”