Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning

By Vishakh Padmakumar, Leonard Lausen, Miguel Ballesteros, Sheng Zha, He He, George Karypis. 2022

23 citations recorded by the source corpus.

Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance. In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning. We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups. However, if the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training at a reduced computational cost.

Subfields: Artificial Intelligence, Natural Language Processing, Reinforcement Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Luke S. Zettlemoyer
3. Mohit Bansal
4. Yi Tay
5. Percy Liang
6. Jianfeng Gao
7. Hannaneh Hajishirzi
8. Graham Neubig

In Natural Language Processing

1. Mohit Bansal
2. Quoc V. Le
3. Yejin Choi
4. Xiang Ren
5. Graham Neubig
6. Yi Tay
7. Luke S. Zettlemoyer
8. Jianfeng Gao

In Reinforcement Learning

1. Xinyun Chen
2. Denny Zhou
3. Jason Wei
4. Xiang Ren
5. Yi Tay
6. Adam Roberts
7. Slav Petrov
8. Shixiang Shane Gu

Citations in the corpus

• Scaling Instruction-Finetuned Language Models (2022, background): “As a more general comparison, our work is among a growing research direction that aims to improve language models without massively scaling the amount of compute (Hoffmann et al., 2022; Padmakumar et al., 2022).”
• Help me write a Poem - Instruction Tuning as a Vehicle for Collaborative Poetry Writing (2022, result): “This result is consistent with prior findings that task-specific finetuning may destroy pretrained representation which leads to degrading performance on other non-finetuning tasks (Aribandi et al., 2021; Padmakumar et al., 2022).”
• Unsupervised Cross-Task Generalization via Retrieval Augmentation (2022, background): “Although there are some existing studies on task-to-task relatedness and transferability (Vu et al., 2020; Lange et al., 2021; Padmakumar et al., 2022), most of them are not designed for unsupervised settings and few are done with multi-task (prompted) upstream models.”