Three new graphical models for statistical language modelling

By Andriy Mnih, Geoffrey E. Hinton. 2007

653 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

GLU Variants Improve Transformer (2020, background): “They also suggest omitting the activation, which they call a "bilinear" layer and attribute to [Mnih and Hinton, 2007].”
Unifying Visual-Semantic Embeddings with Multimodal Neural Language Models (2014, methodology): “We then review log-bilinear neural language models [29], multiplicative neural language models [30] and then introduce our structure-content neural language model.”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “In language domain, several neural network models have been proposed to learn word/n-gram representations [29, 30, 31, 32, 33, 34], sentence representations [35] and paragraph/document representations [36].”
Natural Language Processing (Almost) from Scratch (2011, neutral): “Mnih and Hinton (2007) proposed a related language model approach inspired from Restricted Boltzmann Machines.”
Language Modeling with Gated Convolutional Networks (2016, background): “Another variation of GLUs are bilinear layers (Mnih & Hinton, 2007) which take the form Figure 5 shows that GLUs perform best, followed by bilinear layers and then linear layers.”
Neural Architecture Search with Reinforcement Learning (2016, methodology): “For every convolutional layer, the controller RNN has to select a filter height in [1, 3, 5, 7], a filter width in [1, 3, 5, 7], and a number of filters in [24, 36, 48, 64].”
Tying Word Vectors and Word Classifiers: A Loss Framework for Language Modeling (2016, background): “However, shared input and output representations were implicitly built into these models, rather than proposed as a supplement to a baseline.”
Deep Captioning with Multimodal Recurrent Neural Networks (m-RNN) (2014, methodology): “More closely related to our tasks and method is the work of Kiros et al. (2014b), which is built on a Log-BiLinear model (Mnih & Hinton (2007) 2014)).”
Explain Images with Multimodal Recurrent Neural Networks (2014, methodology): “[18], which is built on a Log-BiLinear model [29] and use CNN to extract visual features.”
A fast and simple algorithm for training neural probabilistic language models (2012, methodology): “For simplicity, we will perform our experiments using the log-bilinear language (LBL) model (Mnih & Hinton, 2007), which is the simplest neural language model.”
Machine Learning Methods That Economists Should Know About (2019, background): “These attempt to capture latent semantic structure in language; see [Mnih and Hinton, 2007, Mnih and Teh, 2012, Mikolov et al., 2013a,b,c, Mnih and Kavukcuoglu, 2013, Pennington et al., 2014, Levy and Goldberg, 2014, Vilnis and McCallum, 2015, Arora et al., 2016, Barkan, 2016, Bamler and Mandt,…”
SHOPPER: A Probabilistic Model of Consumer Choice with Substitutes and Complements (2017, background): “We note that neural language models have spawned myriad developments in other socalled word embedding methods for capturing latent semantic structure in language (Mnih and Hinton, 2007; Mnih and Teh, 2012; Mikolov et al., 2013b,a; Mikolov, Yih and Zweig, 2013; Mnih and Kavukcuoglu, 2013; Pennington,”
Recurrent Convolutional Neural Networks for Text Classification (2015, background): “…(Bengio, Courville, and Vincent 2013) have led to new ideas for solving the data sparsity problem, and many neural models for learning word representations have been proposed (Bengio et al. 2003; Mnih and Hinton 2007; Mikolov 2012; Collobert et al. 2011; Huang et al. 2012; Mikolov et al. 2013).”
Deep Image: Scaling up Image Recognition (2015, neutral): “…many ﬁelds, including visual object recognition (Ciresan et al., 2010), (Krizhevsky et al., 2012), (Szegedy et al., 2014), speech recognition (Dahl et al., 2012), (Han-nun et al., 2014) and natural language processing (Col-lobert & Weston, 2008), (Mnih & Hinton, 2007), (Mnih & Hinton, 2008).”