Atari-HEAD: Atari Human Eye-Tracking and Demonstration Dataset

By Ruohan Zhang, Zhuode Liu, Lin Guan, Luxin Zhang, Mary M. Hayhoe, Dana H. Ballard. 2019

82 citations recorded by the source corpus.

Large-scale public datasets have been shown to benefit research in multiple areas of modern artificial intelligence. For decision-making research that requires human data, high-quality datasets serve as important benchmarks to facilitate the development of new methods by providing a common reproducible standard. Many human decision-making tasks require visual attention to obtain high levels of performance. Therefore, measuring eye movements can provide a rich source of information about the strategies that humans use to solve decision-making tasks. Here, we provide a large-scale, high-quality dataset of human actions with simultaneously recorded eye movements while humans play Atari video games. The dataset consists of 117 hours of gameplay data from a diverse set of 20 games, with 8 million action demonstrations and 328 million gaze samples. We introduce a novel form of gameplay, in which the human plays in a semi-frame-by-frame manner. This leads to near-optimal game decisions and game scores that are comparable or better than known human records. We demonstrate the usefulness of the dataset through two simple applications: predicting human gaze and imitating human demonstrated actions. The quality of the data leads to promising results in both tasks. Moreover, using a learned human gaze model to inform imitation learning leads to an 115% increase in game performance. We interpret these results as highlighting the importance of incorporating human visual attention in models of decision making and demonstrating the value of the current dataset to the research community. We hope that the scale and quality of this dataset can provide more opportunities to researchers in the areas of visual attention, imitation learning, and reinforcement learning.

Subfields: Cognitive Neuroscience, Computer Vision and Pattern Recognition, Human-Computer Interaction

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Computer Vision and Pattern Recognition

In Human-Computer Interaction

Citations in the corpus

Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, background): “Other works use human gaze data as human saliency information, collected with sophisticated eye trackers, to help agents with decision making in an imitation learning setting [50, 52, 14].”
Leveraging Human Guidance for Deep Reinforcement Learning Tasks (2019, background): “Recent works have trained convolutional neural networks to predict the probability distribution of human gaze given a raw image and were able to generate a gaze heatmap in the pixel space [Li et al., 2018; Palazzi et al., 2018; Kim et al., 2018; Zhang et al., 2019].”
Symbols as a Lingua Franca for Bridging Human-AI Chasm for Explainable and Advisable AI Systems (2021, unknown): “While STSTs – be they saliency regions over images, or motion trajectories – have been used in the machine learning community as a means to either advice or interpret the operation of AI systems (Greydanus et al. 2018; Zhang et al. 2020; Christiano et al. 2017), we contend that they will not scale t”
All One Needs to Know about Metaverse: A Complete Survey on Technological Singularity, Virtual Ecosystem, and Research Agenda (2021, background): “Achieving significant milestones of the above two techniques relies on releasing several high-quality body and eyetracking datasets [320]–[323] combined with the recent advancement in deep learning.”
Free-Lunch Saliency via Attention in Atari Agents (2019, methodology): “Unlike all works discussed in Section 2, we suggest a quantitative way to evaluate attention maps produced by the models, using the information from an eye-tracker recently made available in the Atari Human Eye-Tracking and Demonstration Dataset (Atari-HEAD) [45].”
Human Gaze Assisted Artificial Intelligence: A Review (2020, methodology): “It was proposed that human gaze can be used as an auxiliary guiding signal in the imitation learning paradigm [Zhang et al. , 2019a].”
Efficiently Guiding Imitation Learning Algorithms with Human Gaze (2020, methodology): “, 2018], trained separately for each game on the Atari-HEAD dataset [Zhang et al., 2020].”