A Scalable Hierarchical Distributed Language Model

By Andriy Mnih, Geoffrey E. Hinton. 2008

1,026 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “To pretrain word embedding vectors, left-to-right language modeling objectives have been used (Mnih and Hinton, 2009), as well as objectives to discriminate correct from incorrect words in left and right context (Mikolov et al.”
Using the Output Embedding to Improve Language Models (2016, methodology): “Weight tying was previously used in the log-bilinear model of (Mnih and Hinton, 2009), but the decision to use it was not explained, and its effect on the model’s performance was not tested.”
Exploring the Limits of Language Modeling (2016, background): “The resulting model scales independently of vocabulary size – both for training and inference.”
Generating Sequences With Recurrent Neural Networks (2013, background): “…further difficulty is the high computational cost of evaluating all the exponentials during training (although several methods have been to devised make training large softmax layers more efficient, including tree-based models [25, 23], low rank approximations [27] and stochastic derivatives [26]).”
An efficient framework for learning sentence representations (2018, methodology): “This further limits the size of the vocabulary and the model (Variations of the softmax layer such as hierarchical softmax (Mnih & Hinton, 2009), sampling based softmax (Jean et al., 2014) and sub-word representations (Sennrich et al., 2015) can help alleviate this issue).”
Distributed Representations of Words and Phrases and their Compositionality (2013, methodology): “In our work we use a binary Huffman tree, as it assigns short codes to the frequent words which results in fast training.”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “Estimation of the word vectors itself was performed using different model architectures and trained on various corpora [4, 26, 21, 18, 9], and some of the resulting word vectors were made available for future research and comparison2.”
Efficient softmax approximation for GPUs (2016, methodology): “…or of a subset of them (Bengio et al., 2003b; Ji et al., 2015), from (ii) the approaches that compute exact probabilities for an approximate model yielding a lower computational time, such as the popular hierarchical softmax (Goodman, 2001a; Mnih & Hinton, 2009; Morin & Bengio, 2005).”
Exploiting Similarities among Languages for Machine Translation (2013, background): “It is unlike traditional neural network based language models (Bengio et al., 2003; Mnih and Hinton, 2008; Mikolov et al., 2010), where the objective is to predict the next word given the context of several preceding words.”
A fast and simple algorithm for training neural probabilistic language models (2012, methodology): “Unfortunately, the predictive performance of the resulting model is heavily dependent on the tree used and ﬁnding a good tree is a diﬃcult problem (Mnih & Hinton, 2009).”
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2020, background): “For many years, softmax has been the bottleneck for training classification models with a large number of categories (Goodman, 2001; Morin & Bengio, 2005; Mnih & Hinton, 2009).”
Improved Recurrent Neural Networks for Session-based Recommendations (2016, methodology): “Typical approaches include the use of a hierarchical softmax layer [19], and sampling only the most frequent items.”
Bayesian Neural Word Embedding (2016, background): “(3) as follows These biases often explain properties such as frequency of a word in the text [1], [3] or popularity of an item in the dataset [11].”
Word Representations via Gaussian Embedding (2014, background): “Related work in probabilistic matrix factorization (Mnih & Salakhutdinov, 2007) embeds rows and columns as Gaussians, and some forms of this do provide each row and column with its own variance (Salakhutdinov & Mnih, 2008).”
Deep Learning: Methods and Applications (2014, neutral): “This approach is generalized to deeper structures as reported in [253].”
GP-Tree: A Gaussian Process Classifier for Few-Shot Incremental Learning (2021, background): “Morin & Bengio (2005), and following it Mnih & Hinton (2008) proposed a tree-based classifier for language modeling.”
Relevance-based Word Embedding (2017, methodology): “We refer the reader to [25, 26] for the details of calculating the hierarchical softmax approximation.”
Deep Image: Scaling up Image Recognition (2015, neutral): “…many ﬁelds, including visual object recognition (Ciresan et al., 2010), (Krizhevsky et al., 2012), (Szegedy et al., 2014), speech recognition (Dahl et al., 2012), (Han-nun et al., 2014) and natural language processing (Col-lobert & Weston, 2008), (Mnih & Hinton, 2007), (Mnih & Hinton, 2008).”