A Survey on Multi-Task Learning
By Yu Zhang, Qiang Yang. 2017
3,013 citations recorded by the source corpus.
Multi-Task Learning (MTL) is a learning paradigm in machine learning and its aim is to leverage useful information contained in multiple related tasks to help improve the generalization performance of all the tasks. In this paper, we give a survey for MTL from the perspective of algorithmic modeling, applications and theoretical analyses. For algorithmic modeling, we give a definition of MTL and then classify different MTL algorithms into five categories, including feature learning approach, low-rank approach, task clustering approach, task relation learning approach and decomposition approach as well as discussing the characteristics of each approach. In order to improve the performance of learning tasks further, MTL can be combined with other learning paradigms including semi-supervised learning, active learning, unsupervised learning, reinforcement learning, multi-view learning and graphical models. When the number of tasks is large or the data dimensionality is high, we review online, parallel and distributed MTL models as well as dimensionality reduction and feature hashing to reveal their computational and storage advantages. Many real-world applications use MTL to boost their performance and we review representative works in this paper. Finally, we present theoretical analyses and discuss several future directions for MTL.
Subfields: Artificial Intelligence, Optimization & Training, Meta & Continual Learning
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Dacheng Tao
- Yoshua Bengio
- Percy Liang
- Qiang Yang
- Graham Neubig
- Jianfeng Gao
- Sergey Levine
In Optimization & Training
- Jakub Konecný
- Peter Richtárik
- Martin Jaggi
- Jianyu Wang
- Oluwasanmi Koyejo
- Gauri Joshi
- Sebastian U. Stich
- Qiang Yang
In Meta & Continual Learning
- Tinne Tuytelaars
- Marc Masana
- Rahaf Aljundi
- Michael Crawshaw
- Dacheng Tao
- Chelsea Finn
- Joost van de Weijer
- Matthias De Lange
Citations in the corpus
- Advances and Open Problems in Federated Learning (2019, background): “If one considers each client’s local problem (the learning problem on the local dataset) as a separate task (rather than as a shard of a single partitioned dataset), then techniques from multi-task learning [433] immediately become relevant.”
- Fully Decentralized Joint Learning of Personalized Models and Collaboration Graphs (2019, background): “Task relationships are often considered as known a priori but recent work also tries to learn this structure (see Zhang and Yang, 2017, and references therein).”
- Replay in Deep Learning: Current Approaches and Missing Biological Elements (2021, neutral): “…are closely related to the multitask learning literature, where the relationship between task similarity and network performance in terms of accuracy and time has been studied (Zhang & Yang, 2021; Ruder, 2017; Standley et al., 2020; Liu, Johns, & Davison, 2019; Kendall, Gal, & Cipolla, 2018).”
- Continual Lifelong Learning in Natural Language Processing: A Survey (2020, background): “+ positive transfer – negative transfer – task boundaries – off-line learning (Caruana, 1997) (Zhang and Yang, 2017) (Ruder, 2017) (McCann et al.”
- Generalizing from a Few Examples (2019, background): “For a comprehensive introduction of multitask learning, please refer to [144] and [97].”
- Which Tasks Should Be Learned Together in Multi-task Learning? (2019, background): “For another good survey of multi-task learning, see [46].”
- A Survey of Sound Source Localization with Deep Learning Methods (2021, neutral): “We believe these tables can be very useful for a quick search of methods with a given set of characteristics.”
- Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data (2020, background): “MTL has the advantage of computational/storage efficiency (Zhang & Yang, 2017), but training models in a multi-task setting is a balancing act; particularly with datasets that have different: (a) dataset sizes, (b) task difficulty levels, and (c) different types of loss functions.”
- Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunch (2023, background): “The rescale operation is crucial to guarantee the success of DARE, and dropping 30% or 40% delta parameters without rescaling would noticeably lead to worse results.”
- A Continual Learning Survey: Defying Forgetting in Classification Tasks (2019, background): “Continual learning is also referred to as lifelong learning [4], [5], [6], [7], [8], [9], sequential learning [10], [11], [12] or incremental learning [13], [14], [15], [16], [17], [18], [19].”
- Modeling Prosodic Phrasing With Multi-Task Learning in Tacotron-Based TTS (2020, background): “Multi-task learning (MTL) is a learning paradigm that leverages information from multiple related tasks to help improve the overall performance [16].”
- Federated learning with hierarchical clustering of local updates to improve training on non-IID data (2020, neutral): “By considering each local learning objective as a task, multi-task learning [14] has been employed in the distributed learning setting in multiple works [15]–[17].”
- SpeechNet: A Universal Modularized Model for Speech Processing Tasks (2021, methodology): “The universal models trained to solve multiple tasks can benefit from multi-task learning (MTL) (Vandenhende ∗* contribute equally et al., 2020; Zhang and Yang, 2017; Ruder, 2017) which has shown to improve the generalizability and performance of models in NLP and computer vision.”
- Pengi: An Audio Language Model for Audio Tasks (2023, background): “Similar to Multi-Task Learning [64, 11], we hypothesize that learning a shared audio encoder and mapping networks helps Pengi to solve individual tasks better.”
- CH-SIMS: A Chinese Multimodal Sentiment Analysis Dataset with Fine-grained Annotation of Modality (2020, background): “Multi-task learning aims to improve the generalization performance of multiple related tasks by utilizing useful information contained in these tasks (Zhang and Yang, 2017).”
- Recent Advances in Bayesian Optimization (2022, methodology): “To facilitate efficient inference of the work [274], pseudo inputs are adopted to derive a sparse construction for the GP [158].”
- Joint Modelling of Emotion and Abusive Language Detection (2020, methodology): “This affects the stability of training, which may deteriorate performance in comparison to an STL model (Zhang and Yang, 2017).”
- Differential Privacy for Deep and Federated Learning: A Survey (2022, background): “As the multi-task learning paradigm [126], [127] is to leverage useful knowledge in multiple tasks to improve the generalization performance of all tasks, the authors propose that each device in FL learns a task-specific parameter ω i with the objective function f i .”
- Efficiently Identifying Task Groupings for Multi-Task Learning (2021, background): “While recent work has developed new multi-task learning optimization schemes [28, 10, 45, 53, 11, 50], the problem of deciding which tasks should be trained together in the first place is an understudied and complex issue that is often left to human experts [56].”