Deep Transformers with Latent Depth
By Xian Li, Asa Cooper Stickland, Yuqing Tang, X. Kong. 2020
31 citations recorded by the source corpus.
The Transformer model has achieved state-of-the-art performance in many sequence modeling tasks. However, how to leverage model capacity with large or variable depths is still an open challenge. We present a probabilistic framework to automatically learn which layer(s) to use by learning the posterior distributions of layer selection. As an extension of this framework, we propose a novel method to train one shared Transformer network for multilingual machine translation with different layer selection posteriors for each language pair. The proposed method alleviates the vanishing gradient issue and enables stable training of deep Transformers (e.g. 100 layers). We evaluate on WMT English-German machine translation and masked language modeling tasks, where our method outperforms existing approaches for training deeper Transformers. Experiments on multilingual machine translation demonstrate that this approach can effectively leverage increased model capacity and bring universal improvement for both many-to-one and one-to-many translation with diverse language pairs.
Subfields: Computer Vision and Pattern Recognition, Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Quoc V. Le
- Dacheng Tao
- Kyunghyun Cho
- Ming-Hsuan Yang
- Trevor Darrell
- Yi Yang
- Xiaogang Wang
In Model Architectures
- Edouard Grave
- Angela Fan
- Armand Joulin
- Orhan Firat
- Michael Auli
- Ankur Bapna
- Vitaliy Liptchinsky
- Sergey Edunov
Citations in the corpus
- Beyond English-Centric Multilingual Machine Translation (2020, background): “Other work has focused on learning which model components to use (Li et al., 2020) or how to better train language-specific components (Escolano et al.”
- Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference (2021, background): “Most similar to our work in terms of the inference utility is proposed by (Li et al., 2020) where discrete latent variables used to learn language specific layer combinations, whereas in our study we focus on improving inference efficiency of mixture of expert”
- Adaptive Adapters: An Efficient Way to Incorporate BERT Into Neural Machine Translation (2021, methodology): “Inspired by the layer selection method in deep Transformers [52], in this paper, we propose to automatically select and prune adapter layers in inference, to achieve further improvement of the decoding speed and enhance the interpretability of our model.”