Gumbel-Matrix Routing for Flexible Multi-task Learning
By Krzysztof Maziarz, Efi Kokiopoulou, Andrea Gesmundo, Luciano Sbaiz, Gábor Bartók, Jesse Berent. 2019
13 citations recorded by the source corpus.
This paper proposes a novel per-task routing method for multi-task applications. Multi-task neural networks can learn to transfer knowledge across different tasks by using parameter sharing. However, sharing parameters between unrelated tasks can hurt performance. To address this issue, we advocate the use of routing networks to learn flexible parameter sharing, where each group of parameters is shared with a different subset of tasks in order to better leverage tasks relatedness. At the same time, it is known that routing networks are notoriously hard to train. We propose the Gumbel-Matrix routing: a novel multi-task routing method, designed to learn fine-grained patterns of parameter sharing. The routing is learned jointly with the model parameters by standard back-propagation thanks to the Gumbel-Softmax trick. When applied to the Omniglot benchmark, the proposed method reduces the state-of-the-art error rate by 17%.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Optimization & Training
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Henryk Michalewski
- Afroz Mohiuddin
- Xudong Gong
- Hongyan Tang
- Ming Zhao
- Junning Liu
- Aakanksha Chowdhery
- Wojciech Gajewski
In Computer Vision and Pattern Recognition
- Afroz Mohiuddin
- Henryk Michalewski
- Aakanksha Chowdhery
- Wojciech Gajewski
- Lukasz Kaiser
- Luc Van Gool
- Mohit Bansal
- Dacheng Tao
In Model Architectures
- Henryk Michalewski
- Afroz Mohiuddin
- Aakanksha Chowdhery
- Xudong Gong
- Hongyan Tang
- Ming Zhao
- Junning Liu
- Wojciech Gajewski
In Optimization & Training
Citations in the corpus
- Sparse is Enough in Scaling Transformers (2021, methodology): “To determine the active row/column in each block, we reparameterize sampling from a Bernoulli distribution by using the Gumbel-Softmax trick [25].”
- Progressive Layered Extraction (PLE): A Novel Multi-Task Learning (MTL) Model for Personalized Recommendations (2020, methodology): “Similarly, Gumbel-matrix routing framework [15] learns routing of MTL models formulated as a binary matrix with Gumbel-Softmax trick.”