Direct Transfer of Learned Information Among Neural Networks

By L. Pratt, Jack Mostow, Candace A. Kamm. 1991

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning (2021, background): “Transfer learning (Schmidhuber, 1987; Pratt et al., 1991; Caruana et al., 1995) has been the cornerstone of recent progress in natural language processing (Ruder et al., 2019; Devlin et al., 2019; Raffel et al., 2020).”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, background): “…transfer: (1) multitask training (Caruana 1995): training on multiple datasets ( including the target dataset) all at once, (2) sequential training (Pratt, Mostow, and Kamm 1991): ﬁrst training on multiple datasets ( excluding the target dataset) through multitask training, and then continuing to…”
Active Long Term Memory Networks (2016, background): “Generally, in the sequential estimation case, the most recent task benefits from an inductive bias [14, 11] while older tasks become distorted [15, 16, 17] by unconstrained back-propagation [18, 19] of errors in shared parameters, a problem identified as Catastrophic Interferences (CI) between tasks”
A Wholistic View of Continual Learning with Deep Neural Networks: Forgotten Lessons and the Bridge to Active and Open World Learning (2020, background): “The corresponding formulation that is more specific to neural networks is how the representations obtained by learning through backpropagation can be ”recycled” for new tasks [42], [43].”
Meta-Learning in Neural Networks: A Survey (2020, methodology): “TL [35], [56] uses past experience from a source task to improve learning (speed, data efficiency, accuracy) on a target task.”
Inductive biases for deep learning of higher-level cognition (2020, background): “…a fixed data distribution and searching for an inductive bias which works well with this distribution, we are thus interested in transfer learning (Pratt et al., 1991; Pratt, 1993) and continual learning (Ring, 1998) scenarios, with a potentially infinite stream of tasks, and where the learner…”
Extracting comprehensible models from trained neural networks (1996, background): “First, one might want to transfer some part of the solution learned for one task to the process of learning a solution for a related task (Pratt et al., 1991; Caruana, 1996; Thrun, 1996).”
Embodied intelligence via learning and evolution (2021, unknown): “This approach is similar to evaluating the quality of latent neural representations by computing their performance on downstream tasks via transfer learning 33–35 .”
Adapted Deep Embeddings: A Synthesis of Methods for k-Shot Inductive Transfer Learning (2018, background): “Nonetheless, the notion of weight adaptation is extremely popular in deep learning because it can provide a large time savings over training models from scratch, and it may prevent overfitting when the target domain is data constrained [24, 36].”
On the Emergence of Rules in Neural Networks (2002, background): “It has been shown that learning in a new task is significantly accelerated by transferring weights in neural networks without feedback connections if the same input domain is used ( Pratt, Mostow, & Kamm, 1991; Pratt, 1993).”
Meta-neural networks that learn by learning (1992, methodology): “More recent work on using classifiers trained previously on related tasks appears in Pratt’s work [ 3 ].”
Knowledge-Based Artificial Neural Networks (1994, unknown): “(See [41] for a possible transfer method.”
Multitask Learning (1997, background): “The early work on sequential transfer of learned structure between neural nets (Pratt et al. 1991; Pratt 1992; Sharkey & Sharkey 1992) clearly demonstrates that what is learned for one task can be used as a bias for other tasks.”
Phase Transitions in Transfer Learning for High-Dimensional Perceptrons (2021, unknown): “Related Work The idea of transferring informaton between different domains or different tasks was first proposed in [1] and further developed in [2].”
Multi-Domain Adversarial Learning (2019, background): “Since then, transfer learning has come to designate methods to learn across drifting, shifting or distinct distributions, or even distinct tasks (Pratt et al., 1991; Pan & Yang, 2010).”
Extracting refined rules from knowledge-based neural networks (2004, background): “Also, the fruits of training neural networks are difficult to transfer to other neural networks (Pratt et al., (1991) ameliorate this problem), and all but impossible to directly transfer to non-neural learning systems.”