Multi-task Deep Reinforcement Learning with PopArt

By Matteo Hessel, Hubert Soyer, Lasse Espeholt, Wojciech M. Czarnecki, Simon Schmitt, Hado van Hasselt. 2018

363 citations recorded by the source corpus.

The reinforcement learning (RL) community has made great strides in designing algorithms capable of exceeding human performance on specific tasks. These algorithms are mostly trained one task at the time, each new task requiring to train a brand new agent instance. This means the learning algorithm is general, but each solution is not; each agent can only solve the one task it was trained on. In this work, we study the problem of learning to master not one but multiple sequentialdecision tasks at once. A general issue in multi-task learning is that a balance must be found between the needs of multiple tasks competing for the limited resources of a single learning system. Many learning algorithms can get distracted by certain tasks in the set of tasks to solve. Such tasks appear more salient to the learning process, for instance because of the density or magnitude of the in-task rewards. This causes the algorithm to focus on those salient tasks at the expense of generality. We propose to automatically adapt the contribution of each task to the agent’s updates, so that all tasks have a similar impact on the learning dynamics. This resulted in state of the art performance on learning to play all games in a set of 57 diverse Atari games. Excitingly, our method learned a single trained policy - with a single set of weights - that exceeds median human performance. To our knowledge, this was the first time a single agent surpassed human-level performance on this multi-task domain. The same approach also demonstrated state of the art performance on a set of 30 tasks in the 3D reinforcement learning platform DeepMind Lab.

Subfields: Artificial Intelligence, Management Science and Operations Research, Reinforcement Learning, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Reinforcement Learning

In Optimization & Training

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, methodology): “Following Hessel et al. (2019), we augment our agent with multi-task Pop-Art normalization and PixelControl.”
A Survey on Multi-Task Learning (2017, background): “[163] studies the problem of parallel learning of multiple sequential-decision tasks and proposes to automatically adapt the contribution of each task to the updates of the agent to make all tasks have 5600 IEEE TRANSACTIONS ON KNOWLEDGE AND DATA ENGINEERING, VOL.”
Learning World Graphs to Accelerate Hierarchical Reinforcement Learning (2019, neutral): “Finally, the learned world graphs can potentially be applied beyond HRL, e.g., multi-tasking RL [42], structured exploration by using pivotal state as checkouts [25] or in multiagent settings [15, 45].”
Causal Reasoning from Meta-reinforcement Learning (2019, methodology): “…obtained using relatively standard deep RL components – our approach will likely benefit from more advanced architectures (e.g. Hester et al., 2017; Hessel et al., 2018; Espeholt et al., 2018) that allow us to train on longer more complex episodes, as well as models which are more explicitly…”
Continual World: A Robotic Benchmark For Continual Reinforcement Learning (2021, methodology): “We find that using reward normalization as in PopArt [20] is essential to achieve good performance.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “, 2016), pixel control (Jaderberg et al., 2016; Hessel et al., 2019), reward prediction (Jaderberg et al.”
Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learning (2019, methodology): “tudy multi-task learning 2 is the Arcade Learning Environment, a suite of dozens of Atari 2600 games [14]. While having a tremendous impact on the multi-task reinforcement learning research community [9,15,8,16,17], the Atari games included in the benchmark have signiﬁcant differences in visual appe”
TiKick: Towards Playing Multi-agent Football Full Games from Single-agent Demonstrations (2021, positive): “, 2015), PopArt (Hessel et al., 2019), observation normalization, value clipping, layer normalization, and ReLU activation with orthogonal initialization.”
Stabilizing Transformers for Reinforcement Learning (2019, methodology): “The ResNet was adapted from Hessel et al. (2018) and each of its layer blocks consists of a ( 3 × 3 , stride 1) convolution, followed by ( 3 × 3 , stride 2) max-pooling, followed by 2 3 × 3 residual blocks with ReLU non-linearities. where Φ is the standard sinusoid encoding matrix, u ( l ) , v ( l…”
A Generalist Agent (2022, background): “Several poplar RL agents achieve good multi-task RL results within single domains such as Atari57 and DMLab (Espeholt et al., 2018; Song et al., 2020; Hessel et al., 2019).”
Ray Interference: a Source of Plateaus in Deep Reinforcement Learning (2019, neutral): “Additionally, there are methods that prevent the interference directly at the gradient level [5, 47], normalize the scales of the losses [15], or explicitly preserve capacity for late-learned subtasks [18].”
Open-Ended Learning Leads to Generally Capable Agents (2021, methodology): “Similarly to the the original V-MPO implementation, we use single-task PopArt normalisation of the value function (Hessel et al., 2019; van Hasselt et al., 2016).”
V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Control (2019, methodology): “As we are interested in the multi-task setting where a single agent must learn a large number of tasks with differing reward scales, we used PopArt (van Hasselt et al., 2016; Hessel et al., 2018) for the value function, even when training on a single task.”
MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scale (2021, background): “However, to realize these benefits for a real-world robotic learning system, we need to overcome a number of major challenges [64, 32, 11, 86], which have so far made it difficult to produce a large-scale demonstration of multi-task image-based RL that effectively accelerates the acquisition of gene”
Sample Factory: Egocentric 3D Control from Pixels at 100000 FPS with Asynchronous Reinforcement Learning (2020, methodology): “We decided to use a slightly different discretization of the game inputs introduced in (Hessel et al., 2019), since it makes the action space closer to the one available to humans, e.g. it allows the agent to turn and move forward within the same frame.”
The Surprising Effectiveness of MAPPO in Cooperative, Multi-Agent Games (2021, methodology): “To stabilize value learning, we follow the PopArt technique proposed by Hessel et al. (2019) and normalize the values by a running average over the value estimates.”
A Comprehensive Survey on Graph Anomaly Detection With Deep Learning (2021, background): “Opportunities : Multi-task learning provides effective machinery with which incorporate associated tasks [198], [199].”
Attentive Multi-Task Deep Reinforcement Learning (2019, neutral): “ed related work. Glatt et al. [8] train a DQN on a source task and investigate how the learned weights, which are used as initialization for a target task, alter the performance. In a similar manner, [4,6,10] show that some transfer is possible by simply training one network on multiple tasks. Howev”