Learning to Generate Task-Specific Adapters from Task Description

By Qinyuan Ye, Xiang Ren. 2021

36 citations recorded by the source corpus.

Pre-trained text-to-text transformers such as BART have achieved impressive performance across a range of NLP tasks. Recent study further shows that they can learn to generalize to novel tasks, by including task descriptions as part of the source sequence and training the model with (source, target) examples. At test time, these fine-tuned models can make inferences on new tasks using the new task descriptions as part of the input. However, this approach has potential limitations, as the model learns to solve individual (source, target) examples (i.e., at the instance level), instead of learning to solve tasks by taking all examples within a task as a whole (i.e., at the task level). To this end, we introduce Hypter, a framework that improves text-to-text transformer’s generalization ability to unseen tasks by training a hypernetwork to generate task-specific, light-weight adapters from task descriptions. Experiments on ZEST dataset and a synthetic SQuAD dataset demonstrate that Hypter improves upon fine-tuning baselines. Notably, when using BART-Large as the main network, Hypter brings 11.3% comparative improvement on ZEST dataset.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Efficiency & Compression, Meta & Continual Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Hannaneh Hajishirzi
2. Chitta Baral
3. Yejin Choi
4. Swaroop Mishra
5. Daniel Khashabi
6. Noah A. Smith
7. Luke S. Zettlemoyer
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Hannaneh Hajishirzi
2. Chitta Baral
3. Swaroop Mishra
4. Daniel Khashabi
5. Mihir Parmar
6. Yejin Choi
7. Noah A. Smith
8. Yizhong Wang

In Efficiency & Compression

1. Noah A. Smith
2. Yejin Choi
3. Yizhong Wang
4. Daniel Khashabi
5. Arjun Ashok
6. Hannaneh Hajishirzi
7. Giannis Karamanolakis

In Meta & Continual Learning

1. Swaroop Mishra
2. Arjun Ashok
3. Chitta Baral

Citations in the corpus

• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks (2022, background): “Recent literature has been motivated by building models that are generalizable across a variety of NLP tasks, when prompted with either a few examples (Ye and Ren, 2021; Bragg et al., 2021) or language definitions (Efrat and Levy, 2020; Weller et al.”
• Benchmarking Generalization via In-Context Instructions on 1, 600+ Language Tasks (2022, background): “A recent literature is motivated by building models that are generalizable across a variety of language tasks, either when prompted with a few examples (Ye and Ren, 2021; Min et al., 2021; Bragg et al., 2021) or by language definitions (Efrat and Levy, 2020; Weller et al.”