Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learning

By Zhaohan Daniel Guo, B. '. Pires, Bilal Piot, Jean-Bastien Grill, Florent Altch'e, R´emi Munos, Mohammad Gheshlaghi Azar. 2020

158 citations recorded by the source corpus.

Learning a good representation is an essential component for deep reinforcement learning (RL). Representation learning is especially important in multitask and partially observable settings where building a representation of the unknown environment is crucial to solve the tasks. Here we introduce Prediction of Bootstrap Latents (PBL), a simple and flexible self-supervised representation learning algorithm for multitask deep RL. PBL builds on multistep predictive representations of future observations, and focuses on capturing structured information about environment dynamics. Specifically, PBL trains its representation by predicting latent embeddings of future observations. These latent embeddings are themselves trained to be predictive of the aforementioned representations. These predictions form a bootstrapping effect, allowing the agent to learn more about the key aspects of the environment dynamics. In addition, by defining prediction tasks completely in latent space, PBL provides the flexibility of using multimodal observations involving pixel images, language instructions, rewards and more. We show in our experiments that PBL delivers across-the-board improved performance over state of the art deep RL agents in the DMLab-30 and Atari-57 multitask setting.

Subfields: Artificial Intelligence, Reinforcement Learning, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Reinforcement Learning

In Representation & Self-Supervised Learning

Citations in the corpus

Bootstrap Your Own Latent: A New Approach to Self-Supervised Learning (2020, neutral): “Our approach has some similarities with Predictions of Bootstrapped Latents (PBL, [49]), a selfsupervised representation learning technique for reinforcement learning (RL).”
Dynamic Slimmable Network (2021, unknown): “In recent years, a growing number of self-supervised methods with bootstrapping [14, 15, 8] and semi-supervised methods based on consistency regularization [34, 57] use their historical representations to produce targets for the online network.”
How to Train PointGoal Navigation Agents on a (Sample and Compute) Budget (2020, unknown): “Sample efficiency has been studied extensively in model-free RL, with works proposing new training methods [32, 33], auxiliary losses [34, 35, 36], intrinsic rewards [37], pre-training [25], and data-augmentation [38, 39] to improve sample efficiency.”
Model-augmented Prioritized Experience Replay (2022, background): “…et al., 2016) or to enhance representation learning for pixel-based observations (Lin et al., 2019; Agarwal et al., 2021; Stooke et al., 2021; Lee et al., 2020b; Schrittwieser et al., 2020; Hessel et al., 2021; Hafner et al., 2020b; Srinivas et al., 2020; Guo et al., 2020; Lee et al., 2019).”
Bootstrapped Meta-Learning (2021, unknown): “Bootstrapping has recently been introduced in the self-supervised setting [21, 20].”
A Survey on Self-Supervised Learning: Algorithms, Applications, and Future Trends (2023, unknown): “…[145], multi-task learning [157], [158], transfer learning [159], [160], [161], graph NNs [147], [162], [163], reinforcement learning [164], [165], [166], few-shot learning [167], [168], neural architecture search [169], robust learning [146], [170], [171], [172], and meta-learning [173],…”
Data-Efficient Reinforcement Learning with Self-Predictive Representations (2020, unknown): “Moreover, PBL studies multi-task generalization in the asymptotic limits of data, whereas SPR is concerned with single-task performance in low data regimes, using 0 .”
Broaden Your Views for Self-Supervised Video Learning (2021, unknown): “To avoid this, we draw inspiration from recent work [29, 30] in the way we design our networks and losses, as detailed next.”
Contrastive Learning as Goal-Conditioned Reinforcement Learning (2022, background): “This problem is equivalent to a multi-task RL [5, 44, 120, 129, 139], where tasks correspond to reaching goals states.”