Unicorn: Continual Learning with a Universal, Off-policy Agent

By Daniel Jaymin Mankowitz, Augustin Žídek, André Barreto, Dan Horgan, Matteo Hessel, John Quan, Junhyuk Oh, Hado van Hasselt, David Silver, Tom Schaul. 2018

48 citations recorded by the source corpus.

Some real-world domains are best characterized as a single task, but for others this perspective is limiting. Instead, some tasks continually grow in complexity, in tandem with the agent's competence. In continual learning, also referred to as lifelong learning, there are no explicit task boundaries or curricula. As learning agents have become more powerful, continual learning remains one of the frontiers that has resisted quick progress. To test continual learning capabilities we consider a challenging 3D domain with an implicit sequence of tasks and sparse rewards. We propose a novel agent architecture called Unicorn, which demonstrates strong continual learning and outperforms several baseline agents on the proposed domain. The agent achieves this by jointly representing and learning multiple policies efficiently, using a parallel off-policy learning setup.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Reinforcement Learning, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Reinforcement Learning

In Meta & Continual Learning

Citations in the corpus

Off-Policy Actor-Critic with Shared Experience Replay (2019, background): “Finally, it is often useful to learn multiple general value functions (Sutton et al., 2011; Mankowitz et al., 2018; Lample & Chaplot, 2016; Mirowski et al., 2017; Jaderberg et al., 2017b) or options (Sutton et al.”
Towards Continual Reinforcement Learning: A Review and Perspectives (2020, background): “Leveraging such a universal approximator with off-policy learning from many parallel streams of experience in a continual learning fashion is very beneficial in solving tasks with deep dependencies (Mankowitz et al., 2018).”
Continual Learning for Robotics (2019, methodology): “distribution, via either i) setting multiple agents or actors to learn in parallel (Mankowitz et al., 2018), or ii) using a replay buffer (or experience replay (Mnih et al.”
Multi-task Deep Reinforcement Learning with PopArt (2018, background): “Multi-task learning, as considered in this paper, where we get to execute, in parallel, the policies learned for each task, has potential additional beneﬁts, including deep exploration (Osband et al. 2016), and policy composition (Mankowitz et al. 2018; Todorov 2009).”
Hindsight policy gradients (2017, background): “Learning such goal-conditional behavior has received significant attention in machine learning and robotics, especially because a goal-conditional policy may generalize desirable behavior to goals that were never encountered by the agent (Schmidhuber & Huber, 1990; Da Silva et al., 2012; Kupcsik et ”
Online Continual Learning on Sequences (2020, methodology): “• Reinforcement Learning : Arcade Learning Environment (ALE) [3] for Atari games, SURREAL [11] for robot manipulation and RoboTurk for robotic skill learning through imitation [49], CRLMaze extension of VizDoom [39] and DeepMind Lab [50].”
The NetHack Learning Environment (2020, background): “Furthermore, its richness and constant challenge make NetHack an interesting benchmark for lifelong learning [45, 54, 61, 48].”
CURIOUS: Intrinsically Motivated Multi-Task, Multi-Goal Reinforcement Learning (2018, methodology): “In UNICORN, all transitions are replayed with all possible tasks [4].”
Challenges of real-world reinforcement learning: definitions, benchmarks and analysis (2021, background): “Previous research in this area includes learning the options from scratch (Mankowitz et al. 2016a, b; Bacon et al. 2017) as well as planning, given a pre-trained set of options (Schaul et al. 2015; Mankowitz et al. 2018b).”
Discovery of Useful Questions as Auxiliary Tasks (2019, background): “…may also drive the acquisition of a diverse set of skills, even before the main task is known, forming a basis for policy composition or exploration (Andrychowicz et al., 2016; Veeriah et al., 2018; Eysenbach et al., 2018; Florensa et al., 2018; Mankowitz et al., 2018; Riedmiller et al., 2018).”
CURIOUS: Intrinsically Motivated Modular Multi-Goal Reinforcement Learning (2018, unknown): “In UVFA, HER and UNICORN, the next goal to target is selected at random (Schaul et al., 2015a; Andrychowicz et al., 2017; Mankowitz et al., 2018).”
Building a Subspace of Policies for Scalable Continual Learning (2022, background): “There is also a large body of work which leverages the shared structure of the tasks [50, 30, 25, 27, 1, 43], meta-learning [19, 44, 4, 8, 33, 52, 39], or generative models [35, 42, 41, 2] to improve CRL agents, but these are less related to CSP and not very competitive.”
An empirical investigation of the challenges of real-world reinforcement learning (2020, methodology): “Recently, a number of works have surfaced that have shown this formulation to yield robust policies that are agnostic to a range of perturbations in the environment [Tamar et al. 2014; Mankowitz et al. 2018a; Shashua and Mannor 2017; Derman et al. 2018; 2019; Mankowitz et al. 2019].”
Many-Goals Reinforcement Learning (2018, background): “The Unicorn architecture (Mankowitz et al., 2018) and the Intentional Unintentional (IU) agent (Cabi et al.”