HyperGrid: Efficient Multi-Task Transformers with Grid-wise Decomposable Hyper Projections

By Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, Da-Cheng Juan. 2020

9 citations recorded by the source corpus.

Achieving state-of-the-art performance on natural language understanding tasks typically relies on fine-tuning a fresh model for every task. Consequently, this approach leads to a higher overall parameter cost, along with higher technical maintenance for serving multiple models. Learning a single multi-task model that is able to do well for all the tasks has been a challenging and yet attractive proposition. In this paper, we propose \textsc{HyperGrid}, a new approach for highly effective multi-task learning. The proposed approach is based on a decomposable hypernetwork that learns grid-wise projections that help to specialize regions in weight matrices for different tasks. In order to construct the proposed hypernetwork, our method learns the interactions and composition between a global (task-agnostic) state and a local task-specific state. We apply our proposed \textsc{HyperGrid} on the current state-of-the-art T5 model, demonstrating strong performance across the GLUE and SuperGLUE benchmarks when using only a single multi-task model. Our method helps bridge the gap between fine-tuning and multi-task learning approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Denny Zhou
5. Dara Bahri
6. Jason Wei
7. Xuezhi Wang
8. Ed H. Chi

In Computer Vision and Pattern Recognition

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Dara Bahri
5. Denny Zhou
6. Neil Houlsby
7. Zhe Zhao
8. Jai Gupta

In Model Architectures

1. Yi Tay
2. Mostafa Dehghani
3. Donald Metzler
4. Dara Bahri
5. Jai Gupta
6. Neil Houlsby
7. Vinh Q. Tran
8. Jason Wei

Citations in the corpus

• UL2: Unifying Language Learning Paradigms (2022, background): “This has become standard and defacto setup (Raffel et al., 2019; He et al., 2022; Tay et al., 2020; 2021b).”
• Efficient Transformers: A Survey (2020, methodology): “A modest number of models have been proposed, such as PALS (Stickland and Murray, 2019), MAD-X (Pfeiffer et al., 2020) and HyperGrid (Tay et al., 2020c).”
• HyperPrompt: Prompt-based Task-Conditioning of Transformers (2022, methodology): “While Hypernetworks have enjoyed some success in learning adapters (Karimi Mahabadi et al., 2021; Tay et al., 2020) and/or continual learning (von Oswald et al., 2019), we note that this is the first exploration of HyperNetworks as a prompt generator.”