Detecting Attended Visual Targets in Video

By Eunji Chong, Yongxin Wang, Nataniel Ruiz, James M. Rehg. 2020

167 citations recorded by the source corpus.

We address the problem of detecting attention targets in video. Our goal is to identify where each person in each frame of a video is looking, and correctly handle the case where the gaze target is out-of-frame. Our novel architecture models the dynamic interaction between the scene and head features and infers time-varying attention targets. We introduce a new annotated dataset, VideoAttentionTarget, containing complex and dynamic patterns of real-world gaze behavior. Our experiments show that our model can effectively infer dynamic attention in videos. In addition, we apply our predicted attention maps to two social gaze behavior recognition tasks, and show that the resulting classifiers significantly outperform existing methods. We achieve state-of-the-art performance on three datasets: GazeFollow (static images), VideoAttentionTarget (videos), and VideoCoAtt (videos), and obtain the first results for automatically classifying clinically-relevant gaze behavior without wearable cameras or eye trackers.

Subfields: Computer Vision and Pattern Recognition, Pediatrics, Perinatology and Child Health, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, background): “Mutual gaze estimation [54, 150–152, 172, 176] and gaze following [37, 65, 111, 186] are also relevant.”
Dual Attention Guided Gaze Target Detection in the Wild (2021, methodology): “Finally, we ﬁnetune the full model on the VideoAttentionTarget dataset [6].”