An Atari Model Zoo for Analyzing, Visualizing, and Comparing Deep Reinforcement Learning Agents

By Felipe Petroski Such, Vashisht Madhavan, Rosanne Liu, Rui Wang, Pablo Samuel Castro, Yulun Li, Ludwig Schubert, Marc G. Bellemare, Jeff Clune, Joel Lehman. 2018

55 citations recorded by the source corpus.

Much human and computational effort has aimed to improve how deep reinforcement learning (DRL) algorithms perform on benchmarks such as the Atari Learning Environment. Comparatively less effort has focused on understanding what has been learned by such methods, and investigating and comparing the representations learned by different families of DRL algorithms. Sources of friction include the onerous computational requirements, and general logistical and architectural complications for running DRL algorithms at scale. We lessen this friction, by (1) training several algorithms at scale and releasing trained models, (2) integrating with a previous DRL model release, and (3) releasing code that makes it easy for anyone to load, visualize, and analyze such models. This paper introduces the Atari Zoo framework, which contains models trained across benchmark Atari games, in an easy-to-use format, as well as code that implements common modes of analysis and connects such models to a popular neural network visualization library. Further, to demonstrate the potential of this dataset and software package, we show initial quantitative and qualitative comparisons between the performance and representations of several DRL algorithms, highlighting interesting and previously unknown distinctions between them.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentation (2020, methodology): “The oracle uses trained models from Atari Zoo [41] for the Atari games, and a trained DQN model for Pixel-Taxi.”
A Geometric Perspective on Optimal Representations for Reinforcement Learning (2019, background): “The good empirical performance of distributional reinforcement learning (Belle-mare et al., 2017) has also been attributed to representation learning effects, with recent evidence supporting this claim (Such et al., 2018).”
Unsupervised Object Keypoint Learning using Local Spatial Predictability (2020, methodology): “To obtain Atari game frames, we use rollouts of various pre-trained agents in the Atari Model Zoo (Such et al., 2019).”
Observational Overfitting in Reinforcement Learning (2019, background): “Previously, many works interpret the decision-making of an agent through saliency and other network visualizations (Greydanus et al., 2018; Such et al., 2018) on common benchmarks such as Atari.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “However, prior work has found it challenging to uphold this promise [7, 79, 118, 125], prompting many prior methods to employ separate objectives for representation learning and RL [31, 43, 70, 74, 89, 91, 100, 101, 117, 140, 143].”