Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data

By Jonathan Pilault, Amine Elhattami, C. Pal. 2020

105 citations recorded by the source corpus.

Multi-Task Learning (MTL) has emerged as a promising approach for transferring learned knowledge across different tasks. However, multi-task learning must deal with challenges such as: overfitting to low resource tasks, catastrophic forgetting, and negative task transfer, or learning interference. Additionally, in Natural Language Processing (NLP), MTL alone has typically not reached the performance level possible through per-task fine-tuning of pretrained models. However, many fine-tuning approaches are both parameter inefficient, e.g. potentially involving one new model per task, and highly susceptible to losing knowledge acquired during pretraining. We propose a novel transformer based architecture consisting of a new conditional attention mechanism as well as a set of task conditioned modules that facilitate weight sharing. Through this construction we achieve more efficient parameter sharing and mitigate forgetting by keeping half of the weights of a pretrained model fixed. We also use a new multi-task data sampling strategy to mitigate the negative effects of data imbalance across tasks. Using this approach we are able to surpass single-task fine-tuning methods while being parameter and data efficient. With our base model, we attain 2.2% higher performance compared to a full fine-tuned BERT large model on the GLUE benchmark, adding only 5.6% more trained parameters per task (whereas naive fine-tuning potentially adds 100% of the trained parameters per task) and needing only 64.6% of the data. We show that a larger variant of our single multi-task model approach performs competitively across 26 NLP tasks and yields state-of-the-art results on a number of test and development sets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “[39] also proposed a task-conditioned transformer for multi-task learning which is less parameter-efficient.”
VL-ADAPTER: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks (2021, background): “Since we tackle multiple tasks simultaneously [20, 36], we also explore taking advantage of the sharing of information between tasks on adapters and prompts.”
A Survey of Transformers (2021, methodology): “[98] propose a Conditionally Adaptive Multi-Task Learning (CAMTL) framework that uses a trainable attention prior M (zi ) that depends on task encoding zi ∈ RDz”
Natural Language Inference with a Human Touch: Using Human Explanations to Guide Model Attention (2021, methodology): “This baseline out-performs previous state of the art results on SNLI (Zhang et al., 2020; Pilault et al., 2021; Sun et al., 2020), with equivalent performance to a RoBERTa model trained with SNLI, FEVER (Thorne et al., 2018), MNLI, and the ANLI rounds 1, 2 and 3 datasets (Nie et al., 2020).”
Supervising Model Attention with Human Explanations for Robust Natural Language Inference (2021, result): “We ﬁnd that De-BERTa itself achieves 92.59% accuracy, outperforming previous state of the art results on SNLI (Zhang et al. 2020; Pilault, Elhattami, and Pal 2021; Sun et al. 2020).”
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief (2021, background): “, a state-ofthe-art textual entailment engine such as CA-MTL (Pilault et al., 2021).”