Illuminating Generalization in Deep Reinforcement Learning through Procedural Level Generation

By Niels Justesen, Ruben Rodriguez Torrado, Philip Bontrager, Ahmed Khalifa, Julian Togelius, Sebastian Risi. 2018

205 citations recorded by the source corpus.

Deep reinforcement learning (RL) has shown impressive results in a variety of domains, learning directly from high-dimensional sensory streams. However, when neural networks are trained in a fixed environment, such as a single level in a video game, they will usually overfit and fail to generalize to new levels. When RL models overfit, even slight modifications to the environment can result in poor agent performance. This paper explores how procedurally generated levels during training can increase generality. We show that for some games procedural level generation enables generalization to new levels within the same distribution. Additionally, it is possible to achieve better performance with less data by manipulating the difficulty of the levels in response to the performance of the agent. The generality of the learned behaviors is also evaluated on a set of human-designed levels. The results suggest that the ability to generalize to human-designed levels highly depends on the design of the level generators. We apply dimensionality reduction and clustering techniques to visualize the generators' distributions of levels and analyze to what degree they can produce levels similar to those designed by a human.

Subfields: Artificial Intelligence, Automotive Engineering, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Automotive Engineering

In Control and Systems Engineering

Citations in the corpus

Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2018), and by procedurally generating environments (Justesen et al., 2018; Zhang et al., 2018a; Cobbe et al., 2018).”
Quantifying Generalization in Reinforcement Learning (2018, methodology): “(Justesen et al., 2018) use the General Video Game AI (GVG-AI) framework to generate levels from several unique games.”
Leveraging Procedural Generation to Benchmark Reinforcement Learning (2019, background): “In several recent studies (Zhang et al., 2018c; Cobbe et al., 2019; Justesen et al., 2018; Juliani et al., 2019), agents exhibit the capacity to overfit to remarkably large training sets.”
Paired Open-Ended Trailblazer (POET): Endlessly Generating Increasingly Complex and Diverse Learning Environments and Their Solutions (2019, methodology): “This kind of incremental curriculum is intuitive and variants of it appear in the literature when a task is too hard to learn directly [40, 67–70].”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Justesen et al. (2018) train an agent in a procedurally generated environment and update a difficulty parameter based on the agent’s recent success rate – we make use of similar measures to influence task generation in our work.”
The Measure of Intelligence (2019, unknown): “[51, 46, 16, 59, 50]) points to the fact that contemporary Deep Learning models are local-generalization systems, conceptually similar to a locality-sensitive hashtable – they may be trained to achieve arbitrary levels of skill at any task, but doing so requires a dense sampling of the input-cross-t”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “Another vein of research comes from the community attempting to automatically generate content for video games, including training AI agents [80, 154, 82].”
Automatic Data Augmentation for Generalization in Reinforcement Learning (2021, background): “A recent body of work has pointed out the problem of overﬁtting in deep RL [53, 46, 34, 51, 71, 75, 50, 11, 12, 33, 52, 40, 23].”
Increasing generality in machine learning through procedural content generation (2019, methodology): “In work from the PCG research community, Justesen et al. [4] introduced a Progressive PCG approach (PPCG), which showed that performance of training agents can be increased while using less data if the diﬃculty of the level is changed in response to the performance of the agents (Fig.”
Enhanced POET: Open-Ended Reinforcement Learning through Unbounded Invention of Learning Challenges and their Solutions (2020, background): “Interestingly, much research effort is spent attempting to design or learn single-path curricula to help an agent learn a complex task (Bengio et al., 2009; Gomez & Miikkulainen, 1997; Heess et al., 2017; Justesen et al., 2018; Karpathy & Van De Panne, 2012).”
Assessing Generalization in Deep Reinforcement Learning (2018, background): “Both Cobbe et al. (2018) and Justesen et al. (2018) (who also consider video games) use principled procedural generation of training and test levels based on difficulty.”
Domain Generalization with MixStyle (2021, background): “Curriculum learning has been investigated in (Justesen et al., 2018) where the level of training episodes progresses from easy to difﬁcult over the course of training.”
Interference and Generalization in Temporal Difference Learning (2020, methodology): “, 2018), and works proposing to train on a distribution of environments (Zhang et al., 2018c; Cobbe et al., 2018; Justesen et al., 2018) in order to have proper training and test sets (Zhang et al.”
Towards Practical Multi-Object Manipulation using Relational Reinforcement Learning (2019, background): “a new task remains an open challenge [25], [26], [27], [28].”
Automatic Data Augmentation for Generalization in Deep Reinforcement Learning (2020, unknown): “[44, 29, 39, 42, 59, 60, 11, 12, 54, 57, 43].”
Unity: A General Platform for Intelligent Agents (2018, background): “Furthermore, other lines of research argue that procedural generation of environments and measuring success of an algorithm using a train/test split is a principled way of understanding the generalization and robustness of learning algorithms (Cobbe et al., 2019b; Justesen et al., 2018).”
Obstacle Tower: A Generalization Challenge in Vision, Control, and Planning (2019, background): “Having variability in the challenge, ideally through some kind of procedural content generation, is important for avoiding overfitting in reinforcement learning, and being able to evaluate what many AI researchers are actually interested in agent generalization [Cobbe et al., 2018; Zhang et al., 201”