Learning Many Related Tasks at the Same Time with Backpropagation

By R. Caruana. 1994

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

How transferable are features in deep neural networks? (2014, background): “We are interested in the answers to these questions because, to the extent that features within a network are general, we will be able to use them for transfer learning (Caruana, 1995; Bengio et al., 2011; Bengio, 2011).”
Better Mixing via Deep Representations (2012, methodology): “…advantages in terms of statistical power, e.g., as already exploited in prior machine learn-1 ing work such as multi-task learning algorithms (Caruana, 1995; Baxter, 1997; Col-lobert and Weston, 2008) and learning algorithms involving parameter sharing (Lang and Hinton, 1988; LeCun, 1989).”
UNICORN on RAINBOW: A Universal Commonsense Reasoning Model on a New Multitask Benchmark (2021, methodology): “We compare three recipes for intermediate-task transfer: (1) multitask training (Caruana 1995): training on multiple datasets ( including the target dataset) all at once, (2) sequential training (Pratt, Mostow, and Kamm 1991): ﬁrst training on multiple datasets ( excluding the target dataset)…”
True Few-Shot Learning with Language Models (2021, background): “Transfer learning and multitask learning assume access to data that is directly related to the task with limited data [66–69].”
Representation Learning: A Review and New Perspectives (2012, background): “aining classes, and that will be captured by the learned representation, will be useful to model data from other classes or in the context of other tasks. The simplest setting is multi-task learning (Caruana, 1995) and it has been shown in many instances (Caruana, 1995; Collobert and Weston, 2008; C”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, neutral): “Then, the model can be fine-tuned [8, 6, 22].”
Active Long Term Memory Networks (2016, background): “The favorable effect of multi-task learning [12, 13] depends on the shared parametrization of the individual functions and the simultaneous estimation and averaging of multiple losses.”
Meta-Learning: A Survey (2018, neutral): “Neural networks, however, are exceptionally suitable for transfer learning because both the structure and the model parameters of the source models can be used as a good initialization for the target model, yielding a pre-trained model which can then be further fine-tuned using the available trainin”
FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation (2018, background): “The early works (Caruana, 1995; Bengio, 2012; Donahue et al., 2014) apply transfer learning methods to finetune pre-trained models from the common classes containing adequate instances to the uncommon classes with only few instances.”
Learning modular neural network policies for multi-task and multi-robot transfer (2016, background): “An early work by Caruana uses backpropagation to learn many tasks jointly [10].”
Vision-Based Multi-Task Manipulation for Inexpensive Robots Using End-to-End Learning from Demonstration (2017, background): “An early work considered learning a neural network policy on multiple related tasks with backpropagation [23].”
An Ensemble of Epoch-Wise Empirical Bayes for Few-Shot Learning (2019, background): “learningwithveryscarcetraining datamakesitdifﬁculttotrainpowerfulmodelsfornewconcepts. Meta-learning methods aim to tackle this problem by transferring experience from similar few-shot learning tasks [5]. There are different meta strategies, among which the optimization-based methods are particularl”
Few-shot learning of neural networks from scratch by pseudo example optimization (2018, background): “Transfer learning [5] is also widely used for neural network training when a base model trained with a large amount of training data for related tasks is available.”
On the Expressive Power of Deep Architectures (2011, background): “For example, some of the features that are useful for one task may be useful for another, making Deep Learning particularly well suited for transfer learning and multi-task learning (Caruana, 1995; Collobert and Weston, 2008b; Bengio et al., 2011; Bengio, 2011).”
Separating Style and Content (1996, background): “Our work naturally combines two current themes in the connectionist learning literature: factorial learning [7, 3] and learning a family of many related tasks [1, 12]”
A Bayesian/Information Theoretic Model of Learning to Learn via Multiple Task Sampling (1997, background): “However, note that the theoretical model presented here does not apply directly to the experimental results of (Caruana, 1993) because there the training sets are not generated independently for each task.”