HyperPrompt: Prompt-based Task-Conditioning of Transformers
By Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Gupta, Yu Du, Vamsi Krishna Aribandi, Zhe Zhao, Yaguang Li, Zhaoji Chen, Donald Metzler and 2 more. 2022
114 citations recorded by the source corpus.
Prompt-Tuning is a new paradigm for finetuning pre-trained language models in a parameter-efficient way. Here, we explore the use of HyperNetworks to generate hyper-prompts: we propose HyperPrompt, a novel architecture for prompt-based task-conditioning of self-attention in Transformers. The hyper-prompts are end-to-end learnable via generation by a HyperNetwork. HyperPrompt allows the network to learn task-specific feature maps where the hyper-prompts serve as task global memories for the queries to attend to, at the same time enabling flexible information sharing among tasks. We show that HyperPrompt is competitive against strong multi-task learning baselines with as few as $0.14\%$ of additional task-conditioning parameters, achieving great parameter and computational efficiency. Through extensive empirical experiments, we demonstrate that HyperPrompt can achieve superior performances over strong T5 multi-task learning baselines and parameter-efficient adapter variants including Prompt-Tuning and HyperFormer++ on Natural Language Understanding benchmarks of GLUE and SuperGLUE across many model sizes.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yi Tay
- Mostafa Dehghani
- Mohit Bansal
- Donald Metzler
- Denny Zhou
- Luke S. Zettlemoyer
- Jianfeng Gao
- Quoc V. Le
In Computer Vision and Pattern Recognition
- Yi Tay
- Mostafa Dehghani
- Mohit Bansal
- Luc Van Gool
- Dacheng Tao
- Andrew Zisserman
- Donald Metzler
- Trevor Darrell
In Model Architectures
- Yi Tay
- Mostafa Dehghani
- Donald Metzler
- Dara Bahri
- Neil Houlsby
- Mohit Bansal
- Colin Raffel
- Vinh Q. Tran
Citations in the corpus
- UL2: Unifying Language Learning Paradigms (2022, background): “This has become standard and defacto setup (Raffel et al., 2019; He et al., 2022; Tay et al., 2020; 2021b).”
- Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022, background): “[68] recently proposed the use of a hypernetwork to predict prompts for new tasks (rather than training the prompt parameters with gradient descent).”
- Unifying Language Learning Paradigms (2022, methodology): “We use the setup of the latest next generation IR paradigm, i.e., differentiable search index (Tay et al., 2022) for our experiments.”