Reinforcement Learning with Unsupervised Auxiliary Tasks

By Max Jaderberg, Volodymyr Mnih, Wojciech M. Czarnecki, Tom Schaul, Joel Z. Leibo, David Silver, Koray Kavukcuoglu. 2016

1,319 citations recorded by the source corpus.

Deep reinforcement learning agents have achieved state-of-the-art results by directly maximising cumulative reward. However, environments contain a much wider variety of possible training signals. In this paper, we introduce an agent that also maximises many other pseudo-reward functions simultaneously by reinforcement learning. All of these tasks share a common representation that, like unsupervised learning, continues to develop in the absence of extrinsic rewards. We also introduce a novel mechanism for focusing this representation upon extrinsic rewards, so that learning can rapidly adapt to the most relevant aspects of the actual task. Our agent significantly outperforms the previous state-of-the-art on Atari, averaging 880\% expert human performance, and a challenging suite of first-person, three-dimensional \emph{Labyrinth} tasks leading to a mean speedup in learning of 10$\times$ and averaging 87\% expert human performance on Labyrinth.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

Citations in the corpus

Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “Recent work has jointly trained image classifiers with tasks like relative position and colorization (Doersch and Zisserman, 2017), sequence taggers with language modeling (Rei, 2017), and reinforcement learning agents with predicting changes in the environment (Jaderberg et al., 2017).”
Grounded Language Learning in a Simulated 3D World (2017, methodology): “Recently, however, RL agents controlled by deep neural nets have been trained to solve tasks in both 2D (Mnih et al., 2015) and 3D (Mnih et al., 2016) environments.”
What can the brain teach us about building artificial intelligence? (2016, background): “…task is that humans receive feedback on many of these tasks throughout learning, and this both allows more feedback from a single experience (e.g. both an emotional reward for capturing a piece and an explanation of the tactic from a teacher) and constrains the representations that can emerge…”
Emergence of Locomotion Behaviours in Rich Environments (2017, methodology): “Second, like the widely used A3C algorithm [2] and related approaches [3] we distribute the computation over many parallel instances of agent and environment.”
Reward learning from human preferences and demonstrations in Atari (2018, background): “Previous work has shown that training the same highlevel representations on multiple objectives can help performance (Jaderberg et al., 2017).”
Learning to Understand Goal Specifications by Modelling Reward (2018, methodology): “In this setting, we show that the learning speed and performance of A3C agents trained with AGILE reward models is superior to A3C agents trained against environment reward, and comparable to that of true-reward A3C agents supplemented by auxiliary unsupervised reward prediction objectives.”
Unsupervised Question Decomposition for Question Answering (2020, background): “, 2018), where an unsupervised initialization can greatly mitigate the issues of exploring from scratch (Jaderberg et al., 2017).”
Mastering Atari, Go, chess and shogi by planning with a learned model (2019, background): “[19] Max Jaderberg, Volodymyr Mnih, Wojciech Marian Czarnecki, Tom Schaul, Joel Z Leibo, David Silver, and Koray Kavukcuoglu.”
The Reactor: A Sample-Efficient Actor-Critic Architecture (2017, methodology): “Also A3C is an on-policy al-gorithm which does not separate the evaluation policy from the behaviour policy, making it difﬁcult to explore safely.”
Reinforcement Learning with Deep Energy-Based Policies (2017, methodology): “…are desirable for exploration, this exploration is typically attained heuristically, for example by injecting noise (Silver et al., 2014; Lillicrap et al., 2015; Mnih et al., 2015) or initializing a stochastic policy with high entropy (Kakade, 2002; Schulman et al., 2015a; Mnih et al., 2016).”
Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “On DMLab-30 we consider both regular hyper-parameter sweeps and sweeps with population based training (PBT) (Jaderberg et al., 2017a).”
DeepMDP: Learning Continuous Latent Space Models for Representation Learning (2019, background): “Concurrently, a suite of methods combining model-free deep reinforcement learning with auxiliary tasks has shown large beneﬁts on a wide variety of domains (Jaderberg et al., 2016; van den Oord et al., 2018; Mirowski et al., 2017).”
Distributed Prioritized Experience Replay (2018, neutral): “As a result, prioritized experience replay is used in many agents, such as Prioritized Dueling DQN (Wang et al., 2016), UNREAL (Jaderberg et al., 2017), DQfD (Hester et al., 2017), and Rainbow (Hessel et al., 2017).”
Learning and Querying Fast Generative Models for Reinforcement Learning (2018, methodology): “Another approach, presented by Jaderberg et al. (2016), is to add auxiliary prediction losses to the RL training criterion in order to encourage implicit learning of environment dynamics. van Seijen et al. (2017) obtain state of the art performance on MS PACMAN with a model free architecture, but…”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, methodology): “Since the current state-of-the-art methods like A3C (Mnih et al., 2016) or UNREAL (Jaderberg et al., 2017b) can require as much as a billion frames and multiple days to master a single domain, training them on tens of domains at once is too slow to be practical.”
Rainbow: Combining Improvements in Deep Reinforcement Learning (2017, methodology): “In Figure 1 we compare the Rainbow’s performance (measured in terms of the median human normalized score across games) to the corresponding curves for A3C, DQN, DDQN, Prioritized DDQN, Dueling DDQN, Distributional DQN, and Noisy DQN.”
Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents (2017, methodology): “Another approach is to use auxiliary tasks that allow agents to start learning a representation before the first extrinsic reward is observed (Jaderberg et al., 2017); the distributions output by the C51 algorithm of Bellemare, Dabney, and Munos (2017) may be viewed as a particularly meaningful set…”
Value Prediction Network (2017, background): “A lot of variations of DQN have been proposed for learning better state representation [37, 16, 9, 22, 36, 24], including the use of memory-based networks for handling partial observability [9, 22, 24], estimating both state-values and advantage-values as a decomposition of Q-values [37], learning s”
12-in-1: Multi-Task Vision and Language Representation Learning (2019, background): “Advances in multi-task learning have been developed in the context of vision [5,20,32,42,52,53], language [10, 25, 26, 31, 37], and robotics [18, 34, 46].”