Gaze Prediction in Dynamic 360° Immersive Videos

By Yanyu Xu, Yanbing Dong, Junru Wu, Zhengzhong Sun, Zhiru Shi, Jingyi Yu, Shenghua Gao. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2018

313 citations recorded by the source corpus.

This paper explores gaze prediction in dynamic 360° immersive videos, i.e., based on the history scan path and VR contents, we predict where a viewer will look at an upcoming time. To tackle this problem, we first present the large-scale eye-tracking in dynamic VR scene dataset. Our dataset contains 208 360° videos captured in dynamic scenes, and each video is viewed by at least 31 subjects. Our analysis shows that gaze prediction depends on its history scan path and image contents. In terms of the image contents, those salient objects easily attract viewers' attention. On the one hand, the saliency is related to both appearance and motion of the objects. Considering that the saliency measured at different scales is different, we propose to compute saliency maps at different spatial scales: the sub-image patch centered at current gaze point, the sub-image corresponding to the Field of View (FoV), and the panorama image. Then we feed both the saliency maps and the corresponding images into a Convolutional Neural Network (CNN) for feature extraction. Meanwhile, we also use a Long-Short-Term-Memory (LSTM) to encode the history scan path. Then we combine the CNN features and LSTM features for gaze displacement prediction between gaze point at a current time and gaze point at an upcoming time. Extensive experiments validate the effectiveness of our method for gaze prediction in dynamic VR scenes.

Subfields: Computer Vision and Pattern Recognition, Urban Studies, Human-Computer Interaction

View Paper: https://www.semanticscholar.org/paper/cc3bd1659d005ee25c26e3f6aab3c7361f6172b7

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Human Gaze Assisted Artificial Intelligence: A Review (2020, background): “…became available for images [Papadopoulos et al. , 2014; Li et al. , 2014; Xu et al. , 2014; Bylinskii et al. , 2015b; Bylinskii et al. , 2015a; Krafka et al. , 2016], videos [Mathe and Sminchisescu, 2014; Wang et al. , 2018 ] , and 360-degree videos [ Zhang et al. , 2018b; Xu et al. , 2018].”