Multitask Learning

By R. Caruana. 1997

Subfields: Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Representation & Self-Supervised Learning

Citations in the corpus

Multi-task Sequence to Sequence Learning (2015, background): “Such framework has been widely studied by Thrun (1996); Caruana (1997); Evgeniou & Pontil (2004); Ando & Zhang (2005); Argyriou et al. (2007); Kumar & III (2012), among many others.”
U-Net: Machine Reading Comprehension with Unanswerable Questions (2018, methodology): “Multi-task models Different from existing work, we regard the MRC with unanswerable questions as a multi-task learning problem (Caruana 1997) by sharing some meta-knowledge.”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, neutral): “There has been extensive prior work on multi-task learning (Caruana, 1997; Ruder, 2017).”
Fast R-CNN (2015, background): “But it also has the potential to improve results because the tasks inﬂuence each other through a shared representation (the ConvNet) [2].”
Language Models are Few-Shot Learners (2020, background): “ multi-task ﬁne-tuning rather than for in-context learning without weight updates. Another approach to increasing generality and transfer-learning capability in language models is multi-task learning [Car97], which ﬁne-tunes on a mixture of downstream tasks together, rather than separately updating ”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “An alternative approach, called “multitask learning” (Ruder, 2017; Caruana, 1997), is to train the model on multiple tasks at a time.”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, background): “Exploiting commonality across machine learning tasks has a rich history studied under transfer learning (Caruana, 1997; Clark et al., 2019b).”
Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding (2019, methodology): “We extend the knowledge distillation method (Hinton et al., 2015) to the multi-task learning setting (Caruana, 1997; Xu et al., 2018; Collobert et al., 2011; Zhang and Yang, 2017; Liu et al., 2015).”
Multi-Task Deep Neural Networks for Natural Language Understanding (2019, background): “Multi-Task Learning (MTL) is inspired by human learning activities where people often apply the knowledge learned from previous tasks to help learn a new task (Caruana, 1997; Zhang and Yang, 2017).”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “More generally, multitask learning has been successful when models are able to capitalize on relatedness amongst tasks while mitigating interference from dissimilarities [Caruana, 1997].”
Representation Learning Using Multi-Task Deep Neural Networks for Semantic Classification and Information Retrieval (2015, methodology): “Motivated by the success of multi-task learning (Caruana, 1997), we propose in this paper a multi-task DNN approach for representation learning that leverages supervised data from many tasks.”
A Hybrid Neural Network Model for Commonsense Reasoning (2019, methodology): “Therefore, inspired by multi-task learning (Caruana, 1997; Liu et al., 2015, 2019b), we propose a hybrid neural network (HNN) model that combines the strengths of both neural language models and a semantic similarity model.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, methodology): “In this section we investigate the effect of data imbalance across languages on these learning dynamics, particularly through the lens of transfer and interference (Caruana, 1997; Rosenstein et al., 2005).”
Parameter-Efficient Transfer Learning for NLP (2019, background): “Early work shows that sharing network parameters across tasks exploits task regularities, yielding improved performance (Caruana, 1997).”
An Overview of Multi-Task Learning in Deep Neural Networks (2017, methodology): “[Caruana, 1998] deﬁnes two tasks to be similar if they use the same features to make a decision.”
BAM! Born-Again Multi-Task Networks for Natural Language Understanding (2019, background): “Multi-task learning for neural networks in general (Caruana, 1997) and within NLP speciﬁcally (Col-lobert and Weston, 2008; Luong et al., 2016) has been widely studied.”
OPT-IML: Scaling Language Model Instruction Meta Learning through the Lens of Generalization (2022, background): “MTL is a very popular paradigm that improves the generalization performances of a task when combined with related tasks by sharing common parameters or representations (Caruana, 1997; Kumar and Daume III, 2012).”
DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “The second is a shared bottom model [4], where all tasks share the same bottom layers, which are in turn connected to task-specific neural networks.”