Hierarchical Probabilistic Neural Network Language Model

By Frederic Morin, Yoshua Bengio. 2005

1,067 citations recorded by the source corpus.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “The resulting model scales independently of vocabulary size – both for training and inference.”
Distributed Representations of Sentences and Documents (2014, background): “In practice, hierarchical softmax (Morin & Bengio, 2005; Mnih & Hinton, 2008; Mikolov et al., 2013c) is preferred to softmax for fast training.”
Distributed Representations of Words and Phrases and their Compositionality (2013, methodology): “In the context of neural network language models, it was ﬁrst introduced by Morin and Bengio [12].”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…Zaremba et al. (2014); Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang…”
Efficient Estimation of Word Representations in Vector Space (2013, methodology): “However, several practical solutions were proposed for avoiding it; either using hierarchical versions of the softmax [23, 21, 17], or avoiding normalized models completely by using models that are not normalized during training [4, 9].”
Adaptive Input Representations for Neural Language Modeling (2018, background): “The hierarchical softmax does this by introducing latent variables or clusters to simplify normalization (Goodman, 2001; Morin & Bengio, 2005; Mikolov et al., 2011).”
Unsupervised Feature Learning via Non-parametric Instance Discrimination (2018, methodology): “Popular techniques to reduce computation include hierarchical softmax [26], noise-contrastive estimation (NCE) [9], and negative sampling [24].”
Character-Aware Neural Language Models (2015, methodology): “L we employ a hierarchical softmax (Morin and Bengio 2005)—a common strategy for training language models with very large |V| —instead of the usual softmax.”
Efficient softmax approximation for GPUs (2016, methodology): “…or of a subset of them (Bengio et al., 2003b; Ji et al., 2015), from (ii) the approaches that compute exact probabilities for an approximate model yielding a lower computational time, such as the popular hierarchical softmax (Goodman, 2001a; Mnih & Hinton, 2009; Morin & Bengio, 2005).”
Joint Training of a Convolutional Network and a Graphical Model for Human Pose Estimation (2014, methodology): “Joint training of neural-networks and graphical models has been previously reported by Ning et al. [23] for image segmentation, and by various groups in speech and language modeling [6, 22].”
Exploiting Similarities among Languages for Machine Translation (2013, background): “An efﬁcient alternative to the full softmax is the hierarchical soft-max (Morin and Bengio, 2005), which greatly reduces the complexity of computing log p ( w i | w j ) (about logarithmically with respect to the vocabulary size).”
Pointing the Unknown Words (2016, neutral): “The hierarchical softmax (Morin and Bengio, 2005), importance sampling (Bengio and Sen´ecal, 2008; Jean et al., 2014), and the noise contrastive estimation (Gutmann and Hyv¨arinen, 2012; Mnih and Kavukcuoglu, 2013) methods are in the class.”
A fast and simple algorithm for training neural probabilistic language models (2012, background): “Alternatively, the vocabulary can be structured into a tree with words at the leaves, allowing exponentially faster computation of word probabilities and their gradients (Morin & Bengio, 2005).”
Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention (2020, background): “For many years, softmax has been the bottleneck for training classification models with a large number of categories (Goodman, 2001; Morin & Bengio, 2005; Mnih & Hinton, 2009).”
Deep Neural Networks for YouTube Recommendations (2016, methodology): “A popular alternative approach is hierarchical softmax [15], but we weren’t”