Rethinking embedding coupling in pre-trained language models

By Hyung Won Chung, Thibault Févry, Henry Tsai, Melvin Johnson, Sebastian Ruder. 2020

187 citations recorded by the source corpus.

We re-evaluate the standard practice of sharing weights between input and output embeddings in state-of-the-art pre-trained language models. We show that decoupled embeddings provide increased modeling flexibility, allowing us to significantly improve the efficiency of parameter allocation in the input embedding of multilingual models. By reallocating the input embedding parameters in the Transformer layers, we achieve dramatically better performance on standard natural language understanding tasks with the same number of parameters during fine-tuning. We also show that allocating additional capacity to the output embedding provides benefits to the model that persist through the fine-tuning stage even though the output embedding is discarded after pre-training. Our analysis shows that larger output embeddings prevent the model's last layers from overspecializing to the pre-training task and encourage Transformer representations to be more general and more transferable to other tasks and languages. Harnessing these findings, we are able to train models that achieve strong performance on the XTREME benchmark without increasing the number of parameters at the fine-tuning stage.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Representation & Self-Supervised Learning

Citations in the corpus

Learning Compact Metrics for MT (2021, positive): “For the pretrained model F , we use Rem-BERT (Chung et al., 2021), a recently published extension of mBERT (Devlin et al., 2019) pre-trained on 104 languages using a combination of Wikipedia and mC4 (Raffel et al., 2020 Figure 2: Impact of the number of ﬁne-tuning languages on zero-shot…”
Compacter: Efficient Low-Rank Hypercomplex Adapter Layers (2021, background): “Prior work indicates that some of the information captured in pretrained models can be ignored for transfer [21, 22].”
Do Transformer Modifications Transfer Across Implementations and Applications? (2021, background): “Chung et al. (2021) showed the beneﬁts of untying the embeddings for the encoder-only models.”
Charformer: Fast Character Transformers via Gradient-based Subword Tokenization (2021, methodology): “A Note on Comparing Character-level and Subword-based Methods Prior work on efﬁcient methods generally compares models with the same number of parameters (Chung et al., 2021).”
UNKs Everywhere: Adapting Multilingual Language Models to New Scripts (2020, background): “The embedding matrix of large multilingual models makes up around 50% of their entire parameter budget (Chung et al., 2021).”
Dialect-robust Evaluation of Generated Text (2022, methodology): “BLEURT pretrains RemBERT (Chung et al., 2021) on augmented data from Wikipedia and then finetunes on human ratings from WMT corpora.”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “BLEURT BLEURT-20 and BLEURT-21-BETA are obtained by ﬁne-tuning Rebalanced mBERT (Chung et al., 2021) (a multilingual variant of BERT) on a combination of two datasets: previous ratings from the WMT shared, task and generated data.”
XTREME-R: Towards More Challenging and Nuanced Multilingual Evaluation (2021, background): “Rebalanced mBERT (RemBERT; Chung et al., 2021) is a more efficient, scaled-up reparameterization of mBERT.”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “Our primary submission is an ensemble between a COMETKIWI model trained on top of RemBERT (Chung et al., 2021) and the same sequence tagger from the primary submission but using a reference-less input during inference.”
Prompting PaLM for Translation: Assessing Strategies and Performance (2022, methodology): “Our k NN approach for conditioning on the input was pio-neered by Liu et al. (2022b), who used RoBERTa embeddings to identify relevant GPT-3 prompts for sentiment, table-to-text, and QA tasks.”
Sentence-T5: Scalable Sentence Encoders from Pre-trained Text-to-Text Models (2021, background): “Recently, Chung et al. (2021) have shown that larger output embeddings (i.e. larger embedding size) effectively prevent the encoder from over-specializing to the pre-training task, thus making the encoder’s representations more general and more transferable.”
Lifting the Curse of Multilinguality by Pre-training Modular Transformers (2022, background): “Chung et al. (2021) demonstrate that de-136 coupling the input embeddings from the predic-137 tion head improves the performance on a number138 of downstream tasks.”
A Primer on Pretrained Multilingual Language Models (2021, unknown): “Note that the output layer is required only during pretraining and can be discarded during ﬁne-tuning and task-inference - a fact that the Rem-BERT model uses to reduce model size (Chung et al., 2021b).”
Writing System and Speaker Metadata for 2,800+ Language Varieties (2022, background): “At the same time, state-of-the-art pre-trained multilingual models (Conneau et al., 2020; Chung et al., 2021) cover around 100 different languages.”
SemEval-2022 Task 2: Multilingual Idiomaticity Detection and Sentence Embedding (2022, methodology): “approach was taken by MaChAmp (van der Goot, 2022), who used multi-task learning across multiple SemEval tasks (2, 3, 4, 6, 10, 11, 12), pretraining a Rebalanced mBERT (RemBERT) (Chung et al., 2020) model across all of the tasks, then retraining a model for each specific task.”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, methodology): “Based on prior study (Chung et al., 2020a), RemBERT em-perically outperforms XLM-R over multiple multi-6 lingual downstream tasks.”
mGPT: Few-Shot Learners Go Multilingual (2022, unknown): “Massively multilingual LMs—mBERT, XLM-R (Conneau et al., 2020), RemBERT (Chung et al., 2021), mBART (Liu et al., 2020b) and mT5 (Xue et al., 2021)—have now pushed state-of-the-art results on various NLP tasks in multiple languages (Kalyan et al., 2021).”
Weight Tying Biases Token Embeddings Towards the Output Space (2026, unknown): “Chung et al. (2020) established that untying helps but did not explain why.”