Backplay: "Man muss immer umkehren"

By Cinjon Resnick, Roberta Raileanu, Sanyam Kapoor, Alex Peysakhovich, Kyunghyun Cho, Joan Bruna. 2018

52 citations recorded by the source corpus.

Model-free reinforcement learning (RL) requires a large number of trials to learn a good policy, especially in environments with sparse rewards. We explore a method to improve the sample efficiency when we have access to demonstrations. Our approach, Backplay, uses a single demonstration to construct a curriculum for a given task. Rather than starting each training episode in the environment's fixed initial state, we start the agent near the end of the demonstration and move the starting point backwards during the course of training until we reach the initial state. Our contributions are that we analytically characterize the types of environments where Backplay can improve training speed, demonstrate the effectiveness of Backplay both in large grid worlds and a complex four player zero-sum game (Pommerman), and show that Backplay compares favorably to other competitive methods known to improve sample efficiency. This includes reward shaping, behavioral cloning, and reverse curriculum generation.

Subfields: Artificial Intelligence, Management Science and Operations Research

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

Citations in the corpus

ROBOTURK: A Crowdsourcing Platform for Robotic Skill Learning through Imitation (2018, methodology): “This simple method for doing reinforcement learning with demonstrations has been motivated theoretically by [45] and used in practice in prior and concurrent work [46, 47, 39].”
Interactive Fiction Games: A Colossal Adventure (2019, methodology): “load and save game states, enabling planning algorithms Monte-Carlo Tree Search [Coulom, 2007] as well as reinforcement learning approaches that rely on the ability to restore state such as Backplay [Resnick et al., 2018] and GoExplore [Ecoffet et al., 2019]. Jericho additionally provides the option”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, background): “obtain a score greater than or equal to the example trajectory all the way from the initial state. Note that a similar algorithm was discovered independently at around the same time by Resnick et al. [65]. While this approach to robustiﬁcation effectively treats the expert trajectory as a curriculum”
Competitive Experience Replay (2019, background): “Resnick et al. (2018b) and Salimans & Chen (2018) propose to train policies on Pommerman (Resnick et al., 2018a) and the Atari game ‘Montezumas Revenge’ by starting each episode from a different point along a demonstration.”
Pommerman: A Multi-Agent Playground (2018, background): “Instructions and an example for building Docker containers from trained agents can be found in our repository (Resnick et al. 2018b).”
Learning Montezuma's Revenge from a Single Demonstration (2018, background): “After the release of an early version of this work, Resnick et al. [2018] published concurrent work exploring the construction of a curriculum from demonstration states, showing promising results in both single-agent and multi-agent tasks.”
One-Shot High-Fidelity Imitation: Training Large-Scale Deep Nets with RL (2018, background): “This approach has been shown to be very beneficial to RL as well as imitation learning (Resnick et al., 2018; Hosu & Rebedea, 2016; Popov et al., 2017).”