Multitask Learning: A Knowledge-Based Source of Inductive Bias

By R. Caruana. 1993

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Universal Language Model Fine-tuning for Text Classification (2018, background): “…Language Processing (NLP) tasks with real-world applications such as spam, fraud, and bot detection (Jindal and Liu, 2007; Ngai et al., 2011; Chu et al., 2012), emergency response (Caragea et al., 2011), and commercial document classiﬁcation, such as for legal discovery (Roitblat et al., 2010).”
An Overview of Multi-Task Learning in Deep Neural Networks (2017, methodology): “Hard parameter sharing is the most commonly used approach to MTL in neural networks and goes back to [Caruana, 1993].”
Fine-tuned Language Models for Text Classification (2018, background): “…(NLP) tasks with many important real-world applications such as spam, fraud, and bot detection (Jindal and Liu, 2007; Ngai et al., 2011; Chu et al., 2012), emergency response (Caragea et al., 2011), and commercial document classification, such as for legal discovery (Roitblat et al., 2010).”
Deep multi-task learning with low level tasks supervised at lower layers (2016, background): “Multiple-tasks supervision at the outermost layer has a strong tradition in neural net models in vision and elsewhere (Caruana, 1993; Zhang and Zhang, 2014; Yim et al., 2015).”
Latent Multi-Task Architecture Learning (2017, methodology): “…explicitly add inductive bias to the model via the regularizer Ω below, but our model also implicitly learns regularization through multi-task learning (Caruana 1993) mediated by the α parameters, while the β parameters are used to learn the mixture functions f ( · ) , as detailed in the following.”
jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models (2020, background): “…paired with some form of multitask learning. jiant enables a variety of complex training pipelines through simple conﬁguration changes, including multi-task training (Caruana, 1993; Liu et al., 2019a) and pretraining, as well as the sequential ﬁne-tuning approach from STILTs (Phang et al., 2018).”
Domain Adaptation for Statistical Classifiers (2006, background): “This related problem is discussed by Thrun (1996), Caruana (1997) and Baxter (2000), among others.”
How to Fine-Tune BERT for Text Classification? (2019, neutral): “Multi-task learning (Caruana, 1993; Collobert and Weston, 2008) is another relevant direction.”
Domain Adaptation for Visual Applications: A Comprehensive Survey (2017, background): “On the other hand, the relation between DA and domain generalization [151, 254, 97, 94, 155], multi-task learning [26, 76, 178] or few-shot learning [149, 80] is much stronger.”
Modeling Task Relationships in Multi-task Learning with Multi-gate Mixture-of-Experts (2018, methodology): “One of the widely used multi-task learning models is proposed by Caruana [8, 9], which has a shared-bottom model structure, where the bottom hidden layers are shared across tasks.”
Bayesian Multitask Learning with Latent Hierarchies (2009, background): “We consider two related, but distinct tasks: domain adaptation (DA) [4, 1, 7] and multitask learning (MTL) [5, 2].”
Sluice networks: Learning what to share between loosely related tasks (2017, methodology): “Hard parameter sharing was introduced by [6].”
An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models (2019, background): “Multi-Task Learning (MTL) via hard parameter sharing (Caruana, 1993) in neural networks has proven to be effective in many NLP problems (Collobert and Weston, 2008).”
Biological Sequence Modeling with Convolutional Kernel Networks (2017, positive): “Thus, it is natural to design a multitask learning scheme [8] that can take advantage of these common predictive patterns to improve the performance.”
A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks (2018, background): “It is known that a model trained in a multi-task framework can take advantage of inductive transfer between the tasks, achieving a better generalization performance (Caruana 1993).”
Lifelong Learning: A Case Study. (1995, background): “In the context of neural network learning, several researchers have proposed metho ds for learning data representations that are tailored towards the built-in bia s of artificial neural networks [58, 52, 44, 9 , 5]. The basic idea here is the same as in Section 3.3.”