Using the Output Embedding to Improve Language Models

By Ofir Press, Lior Wolf. 2016

832 citations recorded by the source corpus.

We study the topmost weight matrix of neural network language models. We show that this matrix constitutes a valid word embedding. When training language models, we recommend tying the input embedding and this output embedding. We analyze the resulting update rules and show that the tied embedding evolves in a more similar way to the output embedding than to the input embedding in the untied model. We also offer a new method of regularizing the output embedding. Our methods lead to a significant reduction in perplexity, as we are able to show on a variety of neural network language models. Finally, we show that weight tying can reduce the size of neural translation models to less than half of their original size without harming their performance.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Attention is All you Need (2017, methodology): “In our model, we share the same weight matrix between the two embedding layers and the pre-softmax linear transformation, similar to [24].”
A Deep Reinforced Model for Abstractive Summarization (2017, background): “Similar weight-sharing meth-ods have been applied to language modeling (Inan et al., 2016; Press and Wolf, 2016).”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “Our language model follows standard best practices: the input and output embedding layers are tied (Inan et al., 2017; Press and Wolf, 2017), all embedding and hidden layers are set to 512, and we used recurrent dropout (Gal and Ghahra-mani, 2016) on the hidden states and embedding layer.”
MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “We share the embedding and softmax weights of the generator as proposed in Bengio et al. (2003); Press & Wolf (2017); Inan et al. (2017).”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, methodology): “We shared the input and output embedding layers as in Press and Wolf (2017).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg,…”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, methodology): “Token embeddings were tied with the ﬁnal output embedding layer (Inan et al., 2016; Press & Wolf, 2016).”
SpanBERT: Improving Pre-training by Representing and Predicting Spans (2019, background): “SpanBERT sums the loss from both the span boundary and the regular masked language model objectives for each token x i in the masked span ( x s , ..., x e ) , while reusing the input embedding (Press and Wolf, 2017) for the target tokens in both MLM and SBO:”
Cloze-driven Pretraining of Self-attention Networks (2019, background): “and output embeddings in a flat softmax with dimension 1024 (Inan et al., 2016; Press and Wolf, 2017).”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “When the output layer is an adaptive softmax with the same partition of V , d , and k as the adaptive input layer, then we can tie the weights (Inan et al., 2016; Press & Wolf, 2017).”
ALFWorld: Aligning Text and Embodied Environments for Interactive Learning (2020, methodology): “, 2019), and tie output embeddings with input embeddings (Press and Wolf, 2016).”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “The decoder uses learned position embeddings and tied input-output token representations (Press & Wolf, 2017).”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “For the Billion word benchmark we use an adaptive softmax output layer to reduce the computational burden of the large vocabulary (Grave et al., 2016; Press & Wolf, 2017) We train on 32 GPUs with batches of 65K tokens for 975K updates.”
Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time (2022, background): “Finally, as a word of caution for practitioners, we remind readers that many recent language models have tied weights on the output and embedding layers (Press and Wolf, 2017).”
SimVLM: Simple Visual Language Model Pretraining with Weak Supervision (2021, background): “We also share parameters between the embedding and the decoder softmax output layer (Press & Wolf, 2016).”