Progressive Neural Networks

By Andrei A. Rusu, Neil C. Rabinowitz, Guillaume Desjardins, Hubert Soyer, James Kirkpatrick, Koray Kavukcuoglu, Razvan Pascanu, Raia Hadsell. 2016

2,948 citations recorded by the source corpus.

Learning to solve complex sequences of tasks--while both leveraging transfer and avoiding catastrophic forgetting--remains a key obstacle to achieving human-level intelligence. The progressive networks approach represents a step forward in this direction: they are immune to forgetting and can leverage prior knowledge via lateral connections to previously learned features. We evaluate this architecture extensively on a wide variety of reinforcement learning tasks (Atari and 3D maze games), and show that it outperforms common baselines based on pretraining and finetuning. Using a novel sensitivity measure, we demonstrate that transfer occurs at both low-level sensory and high-level control layers of the learned policy.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Representation & Self-Supervised Learning

Citations in the corpus

Domain randomization for transferring deep neural networks from simulation to the real world (2017, background): “Researchers in the reinforcement learning community have also studied the problem of domain adaptation by learning invariant feature representations [13], adapting pretrained networks [35], and other methods.”
RL$^2$: Fast Reinforcement Learning via Slow Reinforcement Learning (2016, background): “There has been a growing interest in using deep neural networks for multi-task learning and transfer learning (Parisotto et al., 2015; Rusu et al., 2015; 2016a; Devin et al., 2016; Rusu et al., 2016b).”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “…(Nichol et al., 2018; Baydin et al., 2017), model parameters (Ha et al., 2016a; Platanios et al., 2018) and models that learn new tasks with high sample ef-ﬁciency (Finn et al., 2017) without forgetting existing tasks or languages (Rusu et al., 2016; Kirk-patrick et al., 2017; Lakew et al., 2018).”
Parameter-Efficient Transfer Learning for NLP (2019, background): “Progressive Networks avoid forgetting by instantiating a new network “column” for each task (Rusu et al., 2016).”
Growing a Brain: Fine-Tuning by Increasing Model Capacity (2017, background): “Most relevant to us are the progressive networks of [34], originally proposed for reinforcement learning.”
Toward Understanding Catastrophic Forgetting in Continual Learning (2019, background): “sharing [36, 39] is also another promising direction that can be combined with the above algorithmic solutions to improve continual learning.”
Overcoming catastrophic forgetting in neural networks (2016, background): “Notably, previous reinforcement learning approaches to continual learning have either relied on either adding capacity to the network [Ring, 1998, Rusu et al., 2016] or on learning each task in separate networks, which are then used to train a single network that can play all games[Rusu et al.,…”
Generative Adversarial Networks (2021, neutral): “GANs can create highly realistic images that can be utilized for various purposes, s.a. in video games, virtual reality, and creating Synthetic Data for training ML models.”
One-Shot Imitation Learning (2017, background): “Success stories include domain adaptation in computer vision (Yang et al., 2007; Kulis et al., 2011; Tzeng et al., 2014; Donahue et al., 2014) and control (Tzeng et al., 2015; Rusu et al., 2016; Sadeghi & Levine, 2016; Gupta et al., 2017; Stadie et al., 2017).”
Multi-task Self-Supervised Visual Learning (2017, background): “Inspired by recent works that extract information across network layers for the sake of transfer learning [14, 22, 36], we propose a mechanism which allows a network to choose which layers are fed into each task.”
Learning multiple visual domains with residual adapters (2017, methodology): “The first one [37, 33] is to freeze the network parameters for the old tasks and learn a new task by adding extra parameters.”
IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures (2018, background): “ALE is typically considered a hard multi-task environment, often accompanied by negative transfer between tasks (Rusu et al., 2016).”
A Joint Many-Task Model: Growing a Neural Network for Multiple NLP Tasks (2016, background): “Recently, Rusu et al. (2016) have proposed a progressive neural network model to handle multiple reinforcement learning tasks, such as Atari games.”
Learning without Forgetting (2016, methodology): “Then we either freeze the existing weights and fine-tune the new weights on the new task (“network expansion”), similar to progressive network [18]; or train using Learning without Forgetting”
Experience Replay for Continual Learning (2018, background): “There has recently been a surge of interest in methods for preventing catastrophic forgetting in RL, inspired in part by neuroscience [11, 13, 27, 29].”
A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, neutral): “Except for the TIL setting that always provides the task identity I D t [70], [71], [72], [73], other scenarios generally require the model to determine which θ ( t ) to use based on the input data, as shown in (10).”
Knowledge Flow: Improve Upon Your Teachers (2019, methodology): “Following their setup, we conduct experiments on MNIST, MNIST with digit ‘3’ missing in the training set, CIFAR-100, and ImageNet.”
DisCoRL: Continual Reinforcement Learning via Policy Distillation (2019, methodology): “The third category of strategies, dy-2 namic network architectures , maintains past knowledge thanks to architectural modiﬁcations while learning [Rusu et al., 2016a, Fernando et al., 2017, Li and Hoiem, 2018, Fernando et al., 2017].”
Continual and Multi-Task Architecture Search (2019, methodology): “Others proposed copying the previous task and augmenting with new task’s features (Rusu et al., 2016), intelligent synapses to accumulate task-related information (Zenke et al., 2017), or online variational inference (Nguyen et al., 2017).”