Prioritized Level Replay

By Minqi Jiang, Edward Grefenstette, Tim Rocktaschel. 2020

227 citations recorded by the source corpus.

Simulated environments with procedurally generated content have become popular benchmarks for testing systematic generalization of reinforcement learning agents. Every level in such an environment is algorithmically created, thereby exhibiting a unique configuration of underlying factors of variation, such as layout, positions of entities, asset appearances, or even the rules governing environment transitions. Fixed sets of training levels can be determined to aid comparison and reproducibility, and test levels can be held out to evaluate the generalization and robustness of agents. While prior work samples training levels in a direct way (e.g.~uniformly) for the agent to learn from, we investigate the hypothesis that different levels provide different learning progress for an agent at specific times during training. We introduce Prioritized Level Replay, a general framework for estimating the future learning potential of a level given the current state of the agent's policy. We find that temporal-difference (TD) errors, while previously used to selectively sample past transitions, also prove effective for scoring a level's future learning potential in generating entire episodes that an agent would experience when replaying it. We report significantly improved sample-efficiency and generalization on the majority of Procgen Benchmark environments as well as two challenging MiniGrid environments. Lastly, we present a qualitative analysis showing that Prioritized Level Replay induces an implicit curriculum, taking the agent gradually from easier to harder levels.

Subfields: Artificial Intelligence, Control and Systems Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

Citations in the corpus

Dynamic Dialogue Policy for Continual Reinforcement Learning (2022, background): “Since task ordering is still an open area of research (Jiang et al., 2020), we test three different permutations so that our results do not depend on a specific order.”
Open-Ended Learning Leads to Generally Capable Agents (2021, background): “Similarly, Jiang et al. (2020) propose Prioritised Level Replay which samples new levels to play on based on the observed TD error in recent experience on those levels.”
A Metaverse: Taxonomy, Components, Applications, and Open Challenges (2022, unknown): “Jiang et al. [312] proposed Prioritized Level Replay (PLR), a general framework for selectively sampling the next level of training by prioritizing items that are expected to have higher learning potential upon future revisit.”
CORA: Benchmarks, Baselines, and Metrics as a Platform for Continual Reinforcement Learning Agents (2021, methodology): “For Procgen (Section 6.1), we use the “deep” residual model without an LSTM from the IMPALA paper, following (Jiang et al., 2021; Cobbe et al., 2020).”
Replay-Guided Adversarial Environment Design (2021, methodology): “In contrast, the second method, Prioritized Level Replay (PLR) [17], embodies an alternative form of dynamic curriculum learning that does not assume control of level generation, but instead, the ability to selectively replay existing levels.”
Human-Timescale Adaptation in an Open-Ended Task Space (2023, methodology): “We chose TD-error for our PLR training runs because it has better asymptotic performance in both the zero-shot and the few-shot setting, and because it was shown to perform well in previous work (Jiang et al., 2021a).”
History Compression via Language Models in Reinforcement Learning (2022, methodology): “In accordance with recent work (Jiang et al., 2021; Goyal et al., 2021; Madan et al., 2021), we use PPO as on-policy algorithm for our Minigrid experiments.”
Deep Reinforcement Learning at the Edge of the Statistical Precipice (2021, methodology): “Score distributions which compare PPO [92], MixReg [111], UCB-DrAC [81], PLR [48], PPG [19] and IDAAC [80].”
Dynamic Dialogue Policy Transformer for Continual Reinforcement Learning (2022, background): “Since task ordering is still an open area of research (Jiang et al., 2020), we test three different permutations so that our results do not depend on a specific order.”