A Comparison of learning algorithms on the Arcade Learning Environment

By Aaron Defazio, Thore Graepel. 2014

26 citations recorded by the source corpus.

Reinforcement learning agents have traditionally been evaluated on small toy problems. With advances in computing power and the advent of the Arcade Learning Environment, it is now possible to evaluate algorithms on diverse and difficult problems within a consistent framework. We discuss some challenges posed by the arcade learning environment which do not manifest in simpler environments. We then provide a comparison of model-free, linear learning algorithms on this challenging problem set.

Subfields: Artificial Intelligence, Control and Systems Engineering, Reinforcement Learning, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Marc G. Bellemare
2. Sergey Levine
3. Marlos C. Machado
4. Yuxi Li
5. A. Mahmood
6. Michael H. Bowling
7. Pieter Abbeel
8. Matthew J. Hausknecht

In Control and Systems Engineering

1. Joel Veness
2. Michael H. Bowling
3. Richard S. Sutton
4. Erik Talvitie
5. Harm van Seijen
6. A. Mahmood
7. Patrick M. Pilarski
8. Marc G. Bellemare

In Reinforcement Learning

1. Marc G. Bellemare
2. Sergey Levine
3. Michael H. Bowling
4. Marlos C. Machado
5. Yuxi Li
6. Matthew J. Hausknecht
7. Hado van Hasselt
8. Pieter Abbeel

In Evaluation & Benchmarks

1. Marc G. Bellemare
2. Marlos C. Machado
3. Michael H. Bowling
4. Matthew J. Hausknecht
5. Erik Talvitie
6. Joel Veness
7. Richard S. Sutton
8. Sriram Srinivasan

Citations in the corpus

• Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, methodology): “Defazio and Graepel (2014) have reported divergence when using algorithms such as GQ(λ), without the projection step, and Q-learning.”
• Domain-Independent Optimistic Initialization for Reinforcement Learning (2014, methodology): “Recently the ALE was used to extensively evaluate several different model-free linear learning algorithms, and optimistic initialization had been mentioned as an open problem in this environment (Defazio and Graepel 2014), due to some of the challenges already mentioned here.”
• State of the Art Control of Atari Games Using Shallow Reinforcement Learning (2015, background): “As will be discussed in more detail in Section 6, the DQN experiments exploit nonstandard game-specific prior information performance and also report only one independent trial per game, making it difficult to reproduce these results or to make principled comparisons to other methods.”
• Incremental Off-policy Reinforcement Learning Algorithms (2017, background): “However, these gradient-based TD algorithms are not fully satisfactory as they require the tuning of an extra step-size parameter and are also susceptible to the problem of high variance (Defazio & Graepel 2014).”
• True Online Temporal-Difference Learning (2015, methodology): “RTraces In this final experiment, we compared the performance of true online Sarsa( λ ) with that of accumulate Sarsa( λ ) and replace Sarsa( λ ), on a domain from the Arcade Learning Environment (ALE) (Bellemare et al., 2013; Defazio & Graepel, 2014; Mnih et al., 2015), called Asterix.”
• Off-policy learning based on weighted importance sampling with linear computational complexity (2015, background): “…O ( n ) computational complexity, such as GTD( � ) (Maei 2011), GQ( � ) (Maei & Sutton 2010), and true online GTD( � ) (van Hasselt, Mah-mood & Sutton 2014), all use the ordinary variant of importance sampling and can suffer severely due to the problem of large variance (Defazio & Graepel 2014).”