Learning Montezuma's Revenge from a Single Demonstration

By Tim Salimans, Richard J. Chen. 2018

163 citations recorded by the source corpus.

We propose a new method for learning from a single demonstration to solve hard exploration tasks like the Atari game Montezuma's Revenge. Instead of imitating human demonstrations, as proposed in other recent works, our approach is to maximize rewards directly. Our agent is trained using off-the-shelf reinforcement learning, but starts every episode by resetting to a state from a demonstration. By starting from such demonstration states, the agent requires much less exploration to learn a game compared to when it starts from the beginning of the game at every episode. We analyze reinforcement learning for tasks with sparse rewards in a simple toy environment, where we show that the run-time of standard RL methods scales exponentially in the number of states between rewards. Our method reduces this to quadratic scaling, opening up many tasks that were previously infeasible. We then apply our method to Montezuma's Revenge, for which we present a trained agent achieving a high-score of 74,500, better than any previously published result.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Exploration by Random Network Distillation (2018, methodology): “Signiﬁcant progress has been achieved by methods with access to either expert demonstrations (Pohlen et al., 2018; Aytar et al., 2018; Garmulewicz et al., 2018), special access to the underlying emulator state (Tang et al., 2017; Stanton & Clune, 2018), or both (Salimans & Chen, 2018).”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, methodology): “For this reason, we chose an LfD algorithm that has been shown capable of improving upon its demonstrations: the Backward Algorithm from Salimans and Chen [28].”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, background): “For example, Aytar et al. (2018), Pohlen et al. (2018) and Salimans and Chen (2018b) have shown that RL with demonstrations can address the hard exploration problem in Montezuma’s Revenge.”
Exploration in Deep Reinforcement Learning: A Survey (2022, methodology): “Salimans and Chen [131] single demonstration PPO Imitation with Exploration Strategy Atari: Montezuma Revenge with distraction 74500 (Playing by youtube 41098) Atari images Ac MF D/ D”
BC + RL : Imitation Learning From Non-Optimal Demonstrations (2019, background): “Recent approaches (Riedmiller et al., 2018) have shown advances in dealing with sparse rewards; however, the difficulty suggests the use of imitation learning to aid in exploration such as the usage of reverse curricuums like (Fan et al., 2018) and (Salimans & Chen, 2018).”
Toybox: A Suite of Environments for Experimental Evaluation of Deep Reinforcement Learning (2019, background): “• Reward Function Environments with sparse reward functions are another sticking point for RL, having only recently seen signiﬁcant progress on games in this category (Ecoffet et al., 2018; Salimans & Chen, 2018; Burda et al., 2019).”
First return, then explore (2020, methodology): “While the robustiﬁcation algorithm originates from Salimans & Chen (2018) 29 , it was modiﬁed in various ways (Supplementary Information “The backward algorithm”).”
Competitive Experience Replay (2019, background): “Resnick et al. (2018b) and Salimans & Chen (2018) propose to train policies on Pommerman (Resnick et al., 2018a) and the Atari game ‘Montezumas Revenge’ by starting each episode from a different point along a demonstration.”
Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism (2021, background): “In practice, imitation learning often succeeds with very few samples in contrast to offline RL [41].”
Backplay: "Man muss immer umkehren" (2018, unknown): “The most related work to ours is a blog post describing a method similar to Backplay used to obtain state-of-the-art performance on the challenging Atari game Montezuma’s Revenge (Salimans & Chen, 2018).”
AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence (2019, background): “There is a long history of fruitful research in imitation learning and learning via observation that demonstrates the beneﬁts of exploiting such data [39, 14, 165, 7, 145, 38, 185, 132, 119, 1].”
The NetHack Learning Environment (2020, methodology): “This sparked a long line of research on novel methods for exploration [e.g., 8, 66, 53] and learning from demonstrations [e.g., 31, 62, 6].”