Multi-task learning for natural language processing in the 2020s: where are we going?
By Joseph Worsham, Jugal Kumar Kalita. 2020
93 citations recorded by the source corpus.
Abstract Multi-task learning (MTL) significantly pre-dates the deep learning era, and it has seen a resurgence in the past few years as researchers have been applying MTL to deep learning solutions for natural language tasks. While steady MTL research has always been present, there is a growing interest driven by the impressive successes published in the related fields of transfer learning and pre-training, such as BERT, and the release of new challenge problems, such as GLUE and the NLP Decathlon (decaNLP). These efforts place more focus on how weights are shared across networks, evaluate the re-usability of network components and identify use cases where MTL can significantly outperform single-task solutions. This paper strives to provide a comprehensive survey of the numerous recent MTL contributions to the field of natural language processing and provide a forum to focus efforts on the hardest unsolved problems in the next decade. While novel models that improve performance on NLP benchmarks are continually produced, lasting MTL challenges remain unsolved which could hold the key to better language understanding, knowledge discovery and natural language interfaces.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Luke S. Zettlemoyer
- Quoc V. Le
- Mohit Bansal
- Percy Liang
- Hannaneh Hajishirzi
- Graham Neubig
- Jianfeng Gao
- Jason Wei
In Computer Vision and Pattern Recognition
- Quoc V. Le
- Luke S. Zettlemoyer
- Luc Van Gool
- Dacheng Tao
- Adams Wei Yu
- Hannaneh Hajishirzi
- Trevor Darrell
- Jianfeng Gao
In Natural Language Processing
- Andrew M. Dai
- Quoc V. Le
- Kelvin Guu
- Adams Wei Yu
- Jason Wei
- Mohit Bansal
- Ajit Rajasekharan
- Sivanesan Sangeetha
Citations in the corpus
- Finetuned Language Models Are Zero-Shot Learners (2021, background): “…deep learning (Collobert et al., 2011; Luong et al., 2016; Ruder, 2017; Velay & Daniel, 2018; Clark et al., 2019b; Liu et al., 2019b, inter alia)—see Worsham & Kalita (2020) for a recent survey on MTL for NLP. Differing from prior MTL work which focuses on performance improvements across training…”
- AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processing (2021, background): “The primary focus of MTL can be improving the performance of target tasks with the help of auxiliary tasks or improving the performance of all the tasks [232].”