Neural Probabilistic Language Models

By Yoshua Bengio, Holger Schwenk, Jean-Sébastien Senecal, Frederic Morin, Jean-Luc Gauvain. 2006

593 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Exploring the Limits of Language Modeling (2016, methodology): “Deep Learning and Recurrent Neural Networks (RNNs) have fueled language modeling research in the past years as it allowed researchers to explore many tasks for which the strong conditional independence assumptions are unrealistic.”
Distributed Representations of Sentences and Documents (2014, methodology): “Our technique is inspired by the recent work in learning vector representations of words using neural networks (Bengio et al., 2006; Collobert & Weston, 2008; Mnih & Hinton, 2008; Turian et al., 2010; Mikolov et al., 2013a;c).”
Adversarial Training Methods for Semi-Supervised Text Classification (2016, methodology): “Following Dai and Le [4], we initialized the word embedding matrix and LSTM weights with a pre-trained recurrent language model [3, 20] that was trained on both labeled and unlabeled examples.”
Deep Fragment Embeddings for Bidirectional Image Sentence Mapping (2014, methodology): “In language domain, several neural network models have been proposed to learn word/n-gram representations [28, 29, 30, 31, 32, 33], sentence representations [34] and paragraph/document representations [35].”
Deep Visual-Semantic Alignments for Generating Image Descriptions (2014, neutral): “We then treat these correspondences as training data for our multimodal Recurrent Neural Network model which learns to generate the descriptions.”
Efficient softmax approximation for GPUs (2016, background): “More recently, parametric models based on neural networks have gained popularity for language modeling [3, 22, 33].”
Learning a Recurrent Visual Representation for Image Caption Generation (2014, methodology): “There are numerous papers using recurrent neural networks for language modeling [2, 24, 27, 17].”
Word Representations via Gaussian Embedding (2014, background): “, 2013), word semantics and language modeling (Bengio et al., 2006; Mnih & Hinton, 2008; Mikolov et al., 2013), and many others.”
Zero-Shot Learning by Convex Combination of Semantic Embeddings (2013, methodology): “A more scalable approach to semantic embeddings of class labels builds upon the recent advances in unsupervised neural language modeling [2].”
End-to-end learning of semantic role labeling using recurrent neural networks (2015, methodology): “We trained word embeddings with English Wikipedia (Ewk) corpus using NLM (Bengio et al., 2006).”
Structured Generative Models of Natural Source Code (2014, methodology): “Our models combine natural language processing models of trees, distributed representations of source code elements like in neural probabilistic language models (Bengio et al., 2006), and compiler-like reasoning to enforce code-specific constraints.”
Conversational Contextual Cues: The Case of Personalization and History for Response Ranking (2016, methodology): “We also jointly learn a shared word embedding space (Bengio et al., 2006) and a user embedding space.”
Joint Deep Modeling of Users and Items Using Reviews for Recommendation (2017, background): “Recently, this representation has shown excellent results in many Natural Language Processing (NLP) tasks [7, 4, 21].”
Polyglot: Distributed Word Representations for Multilingual NLP (2013, background): “Recent developments have led to state-of-art performance in several NLP tasks such as language modeling (Bengio et al., 2006; Mikolov et al., 2010), and syntactic tasks such as sequence tagging (Collobert et al.”
A Model of Coherence Based on Distributed Sentence Representation (2014, background): “…connectionist architectures learn a dense, low-dimensional representation of their problem in a hierarchical way that is capable of capturing both semantic and syntactic aspects of tokens (e.g., (Bengio et al., 2006)), entities, N-grams (Wang and Manning, 2012), or phrases (Socher et al., 2013).”
Topic Modeling in Embedding Spaces (2019, background): “Word embeddings eschew the “one-hot” representation of words— a vocabulary-length vector of zeros with a single one—to learn a distributed representation, one where words with similar meanings are close in a lower-dimensional vector space (Rumelhart and Abrahamson, 1973; Bengio et al., 2006).”