A neural probabilistic language model

By Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Janvin. 2003

556 citations recorded by the source corpus.

A goal of statistical language modeling is to learn the joint probability function of sequences of words in a language. This is intrinsically difficult because of the curse of dimensionality: a word sequence on which the model will be tested is likely to be different from all the word sequences seen during training. Traditional but very successful approaches based on n-grams obtain generalization by concatenating very short overlapping sequences seen in the training set. We propose to fight the curse of dimensionality by learning a distributed representation for words which allows each training sentence to inform the model about an exponential number of semantically neighboring sentences. The model learns simultaneously (1) a distributed representation for each word along with (2) the probability function for word sequences, expressed in terms of these representations. Generalization is obtained because a sequence of words that has never been seen before gets high probability if it is made of words that are similar (in the sense of having a nearby representation) to words forming an already seen sentence. Training such large models (with millions of parameters) within a reasonable time is itself a significant challenge. We report on experiments using neural networks for the probability function, showing on two text corpora that the proposed approach significantly improves on state-of-the-art n-gram models, and that the proposed approach allows to take advantage of longer contexts.

Subfields: General Health Professions, Large Language Models, Natural Language Processing, Philosophy

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In General Health Professions

In Large Language Models

In Natural Language Processing

In Philosophy

Citations in the corpus

Distributed Representations of Words and Phrases and their Compositionality (2013, background): “of the earliest use of word representations dates back to 1986 due to Rumelhart, Hinton, and Williams [13]. This idea has since been applied to statistical language modeling with considerable success [1]. The follow up work includes applications to automatic speech recognition and machine translatio”
ERNIE: Enhanced Representation through Knowledge Integration (2019, background): “A very popular model architecture for estimating neural network language model (NNLM) was proposed in (Bengio et al., 2003), where a feed forward neural network with a linear projection layer and a non-linear hidden layer was used to learn the word vector representation.”
The adverse effects of code duplication in machine learning models of code (2018, positive): “guage model. The goal is to show how even relatively simple models are severely impacted by duplication and draw conclusions that generalize to other models. We follow the early work of Bengio et al. [8] for token-level language modeling. Our neural language model (NLM) is described as P(t i)=softma”
Generating Sentences by Editing Prototypes (2017, methodology): “, 2014), neural language models (Bengio et al., 2003) have been widely used due to their flexibility and performance across a wide range of NLP tasks (Kalchbrenner and Blunsom, 2013; Hahn and Mani, 2000; Ritter et al.”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, positive): “ing [48], [49], [50], but we additionally condition these models on images.”
Convolutional Neural Networks for Sentence Classification (2014, background): “…language processing, much of the work with deep learning meth-ods has involved learning word vector representations through neural language models (Bengio et al., 2003; Yih et al., 2011; Mikolov et al., 2013) and performing composition over the learned word vectors for classiﬁcation (Collobert et…”
Bag of Tricks for Efficient Text Classification (2016, background): “Recently, models based on neural networks have become increasingly popular (Kim, 2014; Zhang and LeCun, 2015; Conneau et al., 2016).”
Neural Architecture Search with Reinforcement Learning (2016, methodology): “We also try to combine them with the method of sharing Input and Output embeddings, e.g., Bengio et al. (2003); Mnih & Hinton (2007), especially Inan et al. (2016) and Press & Wolf (2016).”
Efficient softmax approximation for GPUs (2016, methodology): “In particular, importance sampling (Bengio & Sen ´ ecal, 2008; Bengio et al., 2003b) selects a subset of negative targets to approximate the softmax normalization.”
Describing Videos by Exploiting Temporal Structure (2015, background): “ In addition to the representation of the video, its corresponding caption cis also given to the LSTM decoder word by word. The words are represented by the word embedding matrix as suggested ﬁrst in Bengio et al. (2003) where each column corresponds to a different word in the vocabulary. We note as”
Deep Interest Network for Click-Through Rate Prediction (2017, background): “cale of feature and sample becoming larger and larger at the same time. Along with the mining of feature representations, the design of model structure involves more insights. As a pioneer work, NNLM [9] proposes to learn distributed representation for words, aiming to avoid curse of dimensionality ”
ProGen: Language Modeling for Protein Generation (2020, methodology): “We can factorize this distribution using the chain rule of probability (Bengio et al., 2003): This decomposes the problem of conditional protein generation into next-token prediction, where a token x i can either be an amino acid or a conditioning tag.”
Geometric Deep Learning: Going beyond Euclidean data (2016, background): “ure, recently there has been a growing interest in trying to apply learning on non-Euclidean geometric data, for example, in computer graphics and vision [26], [27], [28], natural language processing [29], and biology [30], [31]. The non-Euclidean nature of such data implies that there are no such f”
Large-Scale Hierarchical Text Classification with Recursively Regularized Deep Graph-CNN (2018, background): “, recurrent neural networks (RNNs) [3, 17]1 and convolutional neural networks (CNNs) [8, 25].”
Recurrent Neural Networks for Classifying Relations in Clinical Notes (2017, neutral): “In general domain NLP, a growing number of studies have successfully used recurrent neural networks (RNNs) combined with word embedding [22] on tasks including language modeling [23], text classification [24–27], question answering [25,26,28,29], machine translation [25,30–32], named entity recognit”
BioBERT: a pre-trained biomedical language representation model for biomedical text mining (2019, background): “Training on speciﬁc NLP tasks (e.g., languagemodeling(Bengio et al. ,2003))wherewordrepresentationswere byproductsoftheNLPtasks, ordirectoptimizationofwordrepresentations based on various hypotheses (Mikolov et al. , 2013; Pennington et al. , 2014) was conducted to obtain word representations.”
Gmail Smart Compose: Real-Time Assisted Writing (2019, background): “In recent years with the rapid development in deep learning techniques, a multitude of neural network-based language models have been proposed, significantly improving state-of-the-art performance [3, 26, 36, 37, 40, 49, 51].”