Detection of eye contact with deep neural networks is as accurate as human experts

By Eunji Chong, Elysha Clark-Whitney, A. Southerland, Elizabeth Stubbs, Chanel Miller, Eliana L. Ajodan, Melanie R. Silverman, Catherine Lord, A. Rozga, Rebecca M. Jones and 1 more. 2020

60 citations recorded by the source corpus.

Eye contact is among the most primary means of social communication used by humans. Quantification of eye contact is valuable as a part of the analysis of social roles and communication skills, and for clinical screening. Estimating a subject’s looking direction is a challenging task, but eye contact can be effectively captured by a wearable point-of-view camera which provides a unique viewpoint. While moments of eye contact from this viewpoint can be hand-coded, such a process tends to be laborious and subjective. In this work, we develop a deep neural network model to automatically detect eye contact in egocentric video. It is the first to achieve accuracy equivalent to that of human experts. We train a deep convolutional network using a dataset of 4,339,879 annotated images, consisting of 103 subjects with diverse demographic backgrounds. 57 subjects have a diagnosis of Autism Spectrum Disorder. The network achieves overall precision of 0.936 and recall of 0.943 on 18 validation subjects, and its performance is on par with 10 trained human coders with a mean precision 0.918 and recall 0.946. Our method will be instrumental in gaze behavior analysis by serving as a scalable, objective, and accessible tool for clinicians and researchers. Eye contact is a key social behavior and its measurement could facilitate the diagnosis and treatment of autism. Here the authors show that a deep neural network model can detect eye contact as accurately has human experts.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Radiology, Nuclear Medicine and Imaging, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Radiology, Nuclear Medicine and Imaging

In Human-Computer Interaction

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “The LAM task is most closely related to prior work on eye contact detection in ego-video [36,159], but addresses more diverse and challenging scenarios.”
Detecting Attended Visual Targets in Video (2020, background): “ attention and have developed solutions for speciﬁc settings [43,21,5]. Prior works have also addressed the detection of eye contact and mutual gaze in the context of dementia care [35,39] and autism [53,9,10]. Other work has analyzed mutual gaze in group interactions for inferring rapport [36]. In ”