A Neural Probabilistic Language Model

By Yoshua Bengio, Réjean Ducharme, Pascal Vincent, Christian Janvin. 2003

7,861 citations recorded by the source corpus.

Subfields: Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

In Natural Language Processing

Citations in the corpus

A Structured Self-attentive Sentence Embedding (2017, background): “Much progress has been made in learning semantically meaningful distributed representations of individual words, also known as word embeddings (Bengio et al., 2001; Mikolov et al., 2013).”
Neural Machine Translation in Linear Time (2016, background): “In neural language modelling, a neural network estimates a distribution over sequences of words or characters that belong to a given language (Bengio et al., 2003).”
Sequence to Sequence Learning with Neural Networks (2014, methodology): “[1], who combine an NNLM with a topic model of the input sentence, which improves rescoring performance.”
Neural Machine Translation by Jointly Learning to Align and Translate (2014, methodology): “Since Bengio et al. (2003) introduced a neural probabilistic language model which uses a neural network to model the conditional probability of a word given a ﬁxed number of the preceding words, neural networks have widely been used in machine translation.”
Learning Phrase Representations using RNN Encoder–Decoder for Statistical Machine Translation (2014, methodology): “It has been known for some time that continuous space language models using neural networks are able to learn semantically meaningful embeddings (See, e.g., (Bengio et al., 2003; Mikolov et al., 2013)).”
Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Neural network based language modeling has been heavily researched since its effectiveness was shown by Bengio et al. in [2].”
MaskGAN: Better Text Generation via Filling in the ______ (2018, methodology): “We share the embedding and softmax weights of the generator as proposed in Bengio et al. (2003); Press & Wolf (2017); Inan et al. (2017).”
Supervised Learning of Universal Sentence Representations from Natural Language Inference Data (2017, background): “Distributed representations of words (or word embeddings) (Bengio et al., 2003; Collobert et al., 2011; Mikolov et al., 2013; Pennington et al., 2014) have shown to provide useful features for various tasks in natural language processing and computer vision.”
Semi-supervised sequence tagging with bidirectional language models (2017, background): “Recently, neural LMs (Bengio et al., 2003; Mikolov et al., 2010) have also been integrated in neural machine translation systems (e.”
Semi-supervised Sequence Learning (2015, methodology): “We also find that recurrent language models [2, 24] can be used as a pretraining method for LSTMs.”
GloVe: Global Vectors for Word Representation (2014, background): “For example, Bengio et al. (2003) introduced a model that learns word vector representations as part of a simple neural network architecture for language modeling.”
Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank (2013, positive): “One possibility to remedy this is to use neural word vectors (Bengio et al., 2003).”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “starting from feed forward models (Bengio et al., 2000), recurrent neural networks (Elman, 1990; Mikolov et al.”
Learning Transferable Visual Models From Natural Language Supervision (2021, neutral): “Mikolov, 2014), and language models (Bengio et al., 2003).”
Experience Grounds Language (2020, background): “…and corpora have demonstrated that representations learned from text capture a great deal of information about word meanings in and out of context (Bengio et al., 2003; Collobert and Weston, 2008; Turian et al., 2010; Mikolov et al., 2013; McCann et al., 2017) It has long been acknowledged that…”
Defending Against Neural Fake News (2019, background): “The current state-of-the-art in unconditional text generation views it as a language modeling problem (Bengio et al., 2003), in which the probability of a document x is the product of the conditional probability of generating each token xi given previous tokens: ppxq “ N π”
MASS: Masked Sequence to Sequence Pre-training for Language Generation (2019, methodology): “Actually, the masked language modeling in BERT (Devlin et al., 2018) and the standard language modeling (Bengio et al., 2003; Mikolov et al., 2010) in GPT (Radford et al., 2018) can be viewed as special cases of MASS.”