Neural net language models

By Yoshua Bengio. 2008

125 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Representation Learning: A Review and New Perspectives (2012, methodology): “Convolutional versions of the RBM or its extensions have also been developed (Desjardins and Bengio, 2008; Lee et al., 2009a; Taylor et al., 2010) as well as a probabilistic max-pooling operation built into Convolutional Deep Networks (Lee et al., 2009a,b; Krizhevsky, 2010).”
A semantic matching energy function for learning with multi-relational data (2013, methodology): “The application of SME to WSD is related to work on vector-based models of word meaning (Lund and Burgess 1996; Landauer and Dumais 1997) and neural language models (Bengio 2008; Collobert et al. 2011), in the sense that we learn a vector representation for each lemma to disambiguate.”
Learning Structured Embeddings of Knowledge Bases (2011, methodology): “The embedding idea has been used in NLP via the framework of language models (Bengio et al. 2003; Bengio 2008) where an embedding per word is learnt.”
Practical Recommendations for Gradient-Based Training of Deep Architectures (2012, background): “A patent example is with neural language models (Bengio et al. , 2003; Bengio, 2008), where the input is a sequence of words.”
Deep Learning of Representations: Looking Forward (2013, background): “Supervised and unsupervised objectives can of course be combined (simply added, with a hyper-parameter as coefficient), like in Larochelle and Bengio (2008)’s discriminative RBM.”
Lexicon Infused Phrase Embeddings for Named Entity Resolution (2014, background): “Many different neural language models have been proposed (Bengio et al., 2003; Morin and Bengio, 2005; Bengio, 2008; Mnih and Hinton, 2008; Collobert and Weston, 2008; Mikolov et al., 2010).”
Taxonomy of Risks posed by Language Models (2022, neutral): “LMs aim to capture statistical properties of the language present in their training corpus and can be used to make probabilistic predictions regarding sequences of tokens [17].”
Word Representations: A Simple and General Method for Semi-Supervised Learning (2010, background): “(See Bengio (2008) for a more complete list of references on neural language models.)”
Joint Learning of Words and Meaning Representations for Open-Text Semantic Parsing (2012, methodology): “Our model is designed with the following key concepts: • Named symbolic entities (synsets, relation types, and lemmas) are all associated with a joint ddimensional vector space, termed the " embedding space " , following previous work in neural language models (see (Bengio, 2008)).”
A Comparison of Vector-based Representations for Semantic Composition (2012, methodology): “These include a simple semantic space, where a word’s vector represents its co-occurrence with neighboring words (Mitchell and Lapata, 2010), a syntax-aware space based on weighted distributional tuples that encode typed co-occurrence relations among words (Baroni and Lenci, 2010), and word embeddin”
Large-Scale Learning of Embeddings with Reconstruction Sampling (2011, methodology): “See (Bengio, 2008) for a review in the context of neural-network based models, which are related to the approach described here.”
Structured Embedding via Pairwise Relations and Long-Range Interactions in Knowledge Base (2015, unknown): “Bengio et al. 2006; Bengio 2008).”