Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents

By Marlos C. Machado, Marc G. Bellemare, Erik Talvitie, Joel Veness, Matthew J. Hausknecht, Michael H. Bowling. 2017

623 citations recorded by the source corpus.

The Arcade Learning Environment (ALE) is an evaluation platform that poses the challenge of building AI agents with general competency across dozens of Atari 2600 games. It supports a variety of different problem settings and it has been receiving increasing attention from the scientific community, leading to some high-profile success stories such as the much publicized Deep Q-Networks (DQN). In this article we take a big picture look at how the ALE is being used by the research community. We show how diverse the evaluation methodologies in the ALE have become with time, and highlight some key concerns when evaluating agents in the ALE. We use this discussion to present some methodological best practices and provide new benchmark results using these best practices. To further the progress in the field, we introduce a new version of the ALE that supports multiple game modes and provides a form of stochasticity we call sticky actions. We conclude this big picture look by revisiting challenges posed when the ALE was introduced, summarizing the state-of-the-art in various problems and highlighting problems that remain open.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Evaluation & Benchmarks

Citations in the corpus

Deep Reinforcement Learning that Matters (2017, methodology): “Machado et al. (2017) revisit the Arcade Learning Environment to propose better evaluation methods in these benchmarks.”
OpenSpiel: A Framework for Reinforcement Learning in Games (2019, methodology): “We hope that OpenSpiel could have a similar effect on general RL in games as the Atari Learning Environment [6, 39] has had on single-agent RL.”
Model-Based Reinforcement Learning for Atari (2019, positive): “In this paper, we explore how learned video models can enable learning in the Atari Learning Environment (ALE) benchmark (Bellemare et al., 2015; Machado et al., 2017) with a budget restricted to 100K time steps – roughly to two hours of a play time.”
Surprising Negative Results for Generative Adversarial Tree Search (2018, methodology): “Domain change results: In order to thoroughly test the GDM and RP, we developed a new OpenAI [15] gym-like interface for the latest Arcade Learning Environment (ALE) [32] that supports different modes and difficulties of Atari games.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “ALE), agents have been shown to often resort to memorization of state-action sequences as opposed to achieving true generalization (Machado et al., 2018).”
Continual Reinforcement Learning in 3D Non-stationary Environments (2019, background): “Arcade Learning Environment [29] 210 × 160 × 3 no 60 different games available in the plaform.”
DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correction (2020, background): “, 2018), and use the evaluation protocol with sticky actions (Machado et al., 2018).”
Striving for Simplicity in Off-policy Deep Reinforcement Learning (2019, methodology): “Note that we use the stochastic version of ALE with sticky actions [39] to ward against trajectory overﬁtting.”
QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation (2018, background): “Unlike most reinforcement learning tasks in the literature [13, 14], the primary challenge in this task is not just to maximize reward, but to generalize effectively to previously unseen objects.”
Exploration by Random Network Distillation (2018, methodology): “The suggested training method (Machado et al., 2017) to prevent an agent from simply memorizing the correct sequence of actions is to use sticky actions (i.e. randomly repeating previous action) has not been used in these works.”
RL Unplugged: Benchmarks for Offline Reinforcement Learning (2020, methodology): “Data Description The dataset is generated by running an online DQN agent and recording transitions from its replay during training with sticky actions (Machado et al., 2018).”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, neutral): “is the Arcade Learning Environment, a suite of dozens of Atari 2600 games [14].”
P3O: Policy-on Policy-off Policy Optimization (2019, methodology): “We average the reward over three random seeds to compute the numbers in Table 10 and we follow the evaluation protocol proposed by Machado et al. (2017) to report the results.”
Google Research Football: A Novel Reinforcement Learning Environment (2019, background): “…techniques have been developed to add artiﬁcial randomness to the environment (like skipping a random number of initial frames or using sticky actions), this randomness may still be too structured and easy to predict and incorporate during training (Machado et al. 2018; Hausknecht and Stone 2015).”
The Hanabi Challenge: A New Frontier for AI Research (2019, background): “…play, is learning to play with a set of unknown partners, with only a few games of interaction In the spirit of the proposed evaluation protocols [22] for the Arcade Learning Environment, which discuss recommendations for learning in Atari games, we will make some recommendations on how research…”
Quantifying Generalization in Reinforcement Learning (2018, background): “They ﬁnd that agents have a high capacity to memorize speciﬁc levels in a given training set, and that techniques intended to mitigate overﬁt-ting in RL, including sticky actions (Machado et al., 2018) and random starts (Hausknecht and Stone, 2015), often fail to do so.”
Dopamine: A Research Framework for Deep Reinforcement Learning (2018, methodology): “In Section 4 we revisit the guidelines put forth by Machado et al. (2018) and discuss Dopamine’s adherence to them.”
Behaviour Suite for Reinforcement Learning (2019, background): “This is crucial for scientiﬁc comparisons and something that has become a major problem for many benchmark suites (Machado et al., 2017) (Section 2).”