Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling

By Hakan Inan, Khashayar Khosravi, Richard Socher. 2016

406 citations recorded by the source corpus.

Recurrent neural networks have been very successful at predicting sequences of words in tasks such as language modeling. However, all such models are based on the conventional classification framework, where the model is trained against one-hot targets, and each word is represented both as an input and as an output in isolation. This causes inefficiencies in learning both in terms of utilizing all of the information and in terms of the number of parameters needed to train. We introduce a novel theoretical framework that facilitates better learning in language modeling, and show that our framework leads to tying together the input embedding and the output projection matrices, greatly reducing the number of trainable variables. Our framework leads to state of the art performance on the Penn Treebank with a variety of network models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Optimization & Training

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, background): “Similar weight-sharing meth-ods have been applied to language modeling (Inan et al., 2016; Press and Wolf, 2016).”
Using the Output Embedding to Improve Language Models (2016, methodology): “Independently and concurrently with our work (Inan et al., 2016) presented an explanation for weight tying in NNLMs based on (Hinton et al.”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “tices: the input and output embedding layers are tied (Inan et al., 2017; Press and Wolf, 2017), all embedding and hidden layers are set to 512, and we used recurrent dropout (Gal and Ghahramani, 2016) on the hidden states and embedding layer.”
MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “We share the embedding and softmax weights of the generator as proposed in Bengio et al. (2003); Press & Wolf (2017); Inan et al. (2017).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…Merity et al., 2016; Kalchbrenner et al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005;…”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “When the output layer is an adaptive softmax with the same partition of V , d, and k as the adaptive input layer, then we can tie the weights (Inan et al., 2016; Press & Wolf, 2017).”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “A standard approach is to use a recurrent network to encode the context into a ﬁxed size vector, which is then multiplied by the word embeddings (Inan et al., 2016) using dot product to obtain the logits.”
PALM: Pre-training an Autoencoding&autoregressive Language Model for Context-conditioned Generation (2020, background): “W e is tied with the corresponding part of the input embedding (Inan et al., 2017), and W v and bv are learnable parameters.”
VirTex: Learning Visual Representations from Textual Annotations (2020, background): “However they share the same token embedding matrix (similar to [77]) which is also reused at the output layers of each model (similar to [88, 89]).”
Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021, methodology): “Since the vocabulary is much smaller than WikiText-103’s, we replace the adaptive word embedding and softmax of Baevski & Auli (2018) with a tied word embedding and softmax matrix (Press & Wolf, 2017; Inan et al., 2017).”
Neural Architecture Search with Reinforcement Learning (2016, methodology): “We also try to combine them with the method of sharing Input and Output embeddings, e.g., Bengio et al. (2003); Mnih & Hinton (2007), especially Inan et al. (2016) and Press & Wolf (2016).”
Regularizing and Optimizing LSTM Language Models (2017, background): “Weight tying (Inan et al., 2016; Press & Wolf, 2016) shares the weights between the embedding and softmax layer, substantially reducing the total parameter count in the model.”
Recurrent Highway Networks (2016, background): “9 Variational LSTM + WT + augmented loss (Inan et al., 2016) 51 M 71.”
On the State of the Art of Evaluation in Neural Language Models (2017, background): “Unless otherwise noted, input and output embeddings are shared, see (Inan et al., 2016) and (Press and Wolf, 2016).”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “The second phase trains θ , the parameters of the controller LSTM, for a ﬁxed number of steps, typically set to 2000 in our experiments.”