Playing hard exploration games by watching YouTube

By Yusuf Aytar, Tobias Pfaff, David M. Budden, T. Paine, Ziyun Wang, Nando de Freitas. 2018

286 citations recorded by the source corpus.

Deep reinforcement learning methods traditionally struggle with tasks where environment rewards are particularly sparse. One successful method of guiding exploration in these domains is to imitate trajectories provided by a human demonstrator. However, these demonstrations are typically collected under artificial conditions, i.e. with access to the agent's exact environment setup and the demonstrator's action and reward trajectories. Here we propose a two-stage method that overcomes these limitations by relying on noisy, unaligned footage without access to such data. First, we learn to map unaligned videos from multiple sources to a common representation using self-supervised objectives constructed over both time and modality (i.e. vision and sound). Second, we embed a single YouTube video in this representation to construct a reward function that encourages an agent to imitate human gameplay. This method of one-shot imitation allows our agent to convincingly exceed human-level performance on the infamously hard exploration games Montezuma's Revenge, Pitfall! and Private Eye for the first time, even if the agent is not presented with any environment rewards.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

Citations in the corpus

Scalable agent alignment via reward modeling: a research direction (2018, background): “…learning alone (Hester et al., 2018): even a perfect imitator can only perform as well as the source it is imitating; superhuman performance typically comes from executing human action sequences faster and more reliably by smoothing out inconsistencies in human behavior (Aytar et al., 2018).”
Imitating Interactive Intelligence (2020, background): “Further, it is at present intrinsically difficult to do so because algorithms and systems have not yet been developed that can perceive and understand the intentions of humans in a way that transfers across radical changes in embodiment, environment, and perspective (Stadie et al., 2017; Borsa et al”
RUDDER: Return Decomposition for Delayed Rewards (2018, background): “Using human demonstrations in reinforcement learning led to a huge improvement on some Atari games like Montezuma’s Revenge [93, 2].”
Sample Efficient Adaptive Text-to-Speech (2018, background): “Few-shot learning can also be leveraged in reinforcement learning, such as by imitating human Atari gameplay from a single recorded action sequence (Pohlen et al., 2018) or online video (Aytar et al., 2018).”
Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shift (2019, methodology): “Auxiliary tasks in the form of extra prediction losses have successfully been used to learn better representations and aid the learning of the value function (Jaderberg et al. 2016), (Aytar et al. 2018).”
Exploration by Random Network Distillation (2018, methodology): “Signiﬁcant progress has been achieved by methods with access to either expert demonstrations (Pohlen et al., 2018; Aytar et al., 2018; Garmulewicz et al., 2018), special access to the underlying emulator state (Tang et al., 2017; Stanton & Clune, 2018), or both (Salimans & Chen, 2018).”
Provable Representation Learning for Imitation with Contrastive Fourier Features (2021, background): “Similar contrastive learning objectives have appeared in related works [10, 40], and so our theoretical bounds can be used to explain these previous empirical successes.”
Go-Explore: a New Approach for Hard-Exploration Problems (2019, neutral): “We next test Go-Explore on the harder, more deceptive game of Pitfall, for which all previous RL algorithms scored ≤ 0 points, except those that were evaluated on the fully deterministic version of the game [43, 44] or relied on human demonstrations [26, 27, 45].”
Making Efficient Use of Demonstrations to Solve Hard Exploration Problems (2019, background): “First, the agent has a single learner process which samples from both demonstration and agent buffers in order to update its policy parameters, the pseudocode of the R2D3 learner can be found in Algorithm 1.”
Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observations (2019, methodology): “Aytar et al. (2018) use video demonstrations of experts to learn good policies for the Atari domains of Montezuma’s Revenge, Pitfall, and Private Eye.”
Attentive Multi-Task Deep Reinforcement Learning (2019, methodology): “al the teacher signal such that the student can surpass the teacher’s performance. Further, in [11,7,23] knowledge is transferred from human expert demonstrations, while the algorithm of Aytar et al. [1] learns from YouTube video demonstrations. Gupta et al. [9] transfer knowledge from source to tar”
Temporal Cycle-Consistency Learning (2019, methodology): “Instead of labels, self-supervised learning methods use signals such as temporal order [16, 27], consistency across viewpoints and/or temporal neighbors [35], classifying arbitrary temporal segments [22], temporal distance classification within or across modalities [7], spatial permutation of patche”
Deep reinforcement learning: a survey (2020, background): “Deep meta-RL can also be leveraged in games to do few-shot learning, such as by imitating human Atari game play from a single recorded action sequence (Pohlen et al., 2018) or online videos (Aytar et al., 2018).”
Broadly-Exploring, Local-Policy Trees for Long-Horizon Task Planning (2020, background): “As the temporal distance is not easily quantiﬁable, we learn to predict it with a temporal distance classiﬁer, τ tdc ( x i , x j ) , with K exponential categories corresponding to temporal distance intervals, [2 , 4) , [4 , 8) , [8 , 16) , ..., [128 , 256) } based on [38].”
Two-Timescale Networks for Nonlinear Value Function Approximation (2019, background): “, 2016) or classifying the temporal distance between frames (Aytar et al., 2018).”
DexMV: Imitation Learning for Dexterous Manipulation from Human Videos (2021, background): “The way to perform imitation learning is not limited to behavior cloning [54, 7, 62, 10, 82], but also via inverse reinforcement learning (IRL) [64, 45, 1, 28, 20, 4, 83, 41] and augmenting the RL training data with demonstrations [53, 19, 84, 59, 57].”
“Good Robot!”: Efficient Reinforcement Learning for Multi-Step Visual Tasks with Sim to Real Transfer (2019, methodology): “When available, demonstration can be an effective method for guiding exploration [27], [28], [29].”
The NetHack Learning Environment (2020, background): “, 2017) and learning from demonstrations (e.g. Hester et al., 2017; Salimans & Chen, 2018; Aytar et al., 2018) in Deep RL.”