Agent57: Outperforming the Atari Human Benchmark

By Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Daniel Guo, Charles Blundell. 2020

594 citations recorded by the source corpus.

Atari games have been a long-standing benchmark in the reinforcement learning (RL) community for the past decade. This benchmark was proposed to test general competency of RL algorithms. Previous work has achieved good average performance by doing outstandingly well on many games of the set, but very poorly in several of the most challenging games. We propose Agent57, the first deep RL agent that outperforms the standard human benchmark on all 57 Atari games. To achieve this result, we train a neural network which parameterizes a family of policies ranging from very exploratory to purely exploitative. We propose an adaptive mechanism to choose which policy to prioritize throughout the training process. Additionally, we utilize a novel parameterization of the architecture that allows for more consistent and stable learning.

Subfields: Artificial Intelligence, Automotive Engineering, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Management Science and Operations Research

Citations in the corpus

Celebrating Diversity in Shared Multi-Agent Reinforcement Learning (2021, methodology): “Existing methods such as curiosity-driven algorithms [41–44] or maximising mutual information [45–47] have shown great promise.”
BeBold: Exploration Beyond the Boundary of Explored Regions (2020, background): “Deep reinforcement learning (RL) has experienced signiﬁcant progress over the last several years, with impressive performance in games like Atari (Mnih et al., 2015; Badia et al., 2020a), Star-Craft (Vinyals et al., 2019) and Chess (Silver et al., 2016; 2017; 2018).”
Data-Efficient Reinforcement Learning with Self-Predictive Representations (2020, background): “Deep RL involves training agents based on large neural networks using large amounts of data (Sutton, 2019), a trend evident across both model-based (Schrittwieser et al., 2020) and model-free (Badia et al., 2020) learning.”
Carbontracker: Tracking and Predicting the Carbon Footprint of Training Deep Learning Models (2020, result): “These models achieve impressive results across a wide variety of tasks such as gameplay, where AlphaStar reached the highest rank in the strategy game Starcraft II (Vinyals et al., 2019) and Agent57 surpassed human performance in all 57 Atari 2600 games (Badia et al., 2020).”
Neural Production Systems (2021, background): “havioural priors can drive the necessary exploration (Goyal et al., 2019a; Tirumala et al., 2020; Badia et al., 2020).”
Mastering Atari with Discrete World Models (2020, methodology): “Experimental setup We select the 55 games that prior works in the literature from different research labs tend to agree on (Mnih et al., 2016; Brockman et al., 2016; Hessel et al., 2018; Castro et al., 2018; Badia et al., 2020) and recommend this set of games for evaluation going forward.”
Rethinking Formal Models of Partially Observable Multiagent Decision Making (2019, background): “If the generalization were straightforward, the situation in imperfect-information games would by now likely be closer to that in perfect-information games or single-agent RL, where many of the state-of-the-art algorithms are very general [39, 36, 1].”
A Framework for Efficient Robotic Manipulation (2020, background): “Notable successes include learning to solve a diverse set of challenging video games [1]–[4], mastering complex classical games like Go, Chess, Shogi, and Hanabi [5]–[7], and learning autonomous robotic control policies in both simulated [8]–[11] and real-world settings [12], [13].”
Why Should I Trust You, Bellman? The Bellman Error is a Poor Replacement for Value Error (2022, background): “Consequently, many modern RL algorithms rely on a value function in some capacity (Mnih et al., 2015; Gu et al., 2016; Schulman et al., 2017; Fujimoto et al., 2018; Badia et al., 2020).”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “• Difficulty Progress: One might be more interested in evaluating progress on the hardest tasks on a benchmark [3].”
Effective Diversity in Population-Based Reinforcement Learning (2020, methodology): “…this adaptive mechanism could also be used for count-based exploration methods or intrinsic rewards (Schmidhuber, 2010), and note very recent work using a similar approach to vary exploration in off-policy methods (Schaul et al., 2019; Badia et al., 2020) and model-based RL (Ball et al., 2020).”
Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos (2022, background): “[15] Adrià Puigdomènech Badia, Bilal Piot, Steven Kapturowski, Pablo Sprechmann, Alex Vitvitskyi, Zhaohan Daniel Guo, and Charles Blundell.”
Deep Hierarchical Planning from Pixels (2022, background): “Artiﬁcial agents have achieved remarkable performance on reactive video games (Mnih et al., 2015; Badia et al., 2020) or board games that last for a few hundred moves (Silver et al., 2017).”
Unity: A General Platform for Intelligent Agents (2018, methodology): “Fast & Distributed Simulation - Depending on the sample eﬃciency of the method used, modern machine learning algorithms often require up to billions of samples in order to converge to an optimal solution (Espeholt et al., 2018; Puigdomènech Badia et al., 2020).”
On the link between conscious function and general intelligence in humans and machines (2022, methodology): “Even state-of-the-art AI systems for image recognition or game playing often require multiple orders of magnitude more prior experience or pre-deﬁned domain knowledge in order to solve these tasks than do humans or even other animals (Badia et al., 2020; Dai et al., 2021).”