Factored Neural Language Models

By Andrei Alexandrescu, Katrin Kirchhoff. 2006

108 citations recorded by the source corpus.

We present a new type of neural probabilistic language model that learns a mapping from both words and explicit word features into a continuous space that is then used for word prediction. Additionally, we investigate several ways of deriving continuous word representations for unknown words from those of known words. The resulting model significantly reduces perplexity on sparse-data tasks when compared to standard backoff models, standard neural language models, and factored language models.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Enriching Word Vectors with Subword Information (2016, methodology): “To model rare words better, Alexandrescu and Kirchhoff (2006) introduced factored neural language models, where words are represented as sets of features.”
Character-Aware Neural Language Models (2015, methodology): “In order to address the rare word problem, Alexandrescu and Kirchhoff (2006)—building on analogous work on count-based n-gram language models by Bilmes and Kirchhoff (2003)—represent a word as a set of shared factor embeddings.”
Charagram: Embedding Words and Sentences via Character n-grams (2016, background): “Some added knowledge-based morphological features to word representations (Alexandrescu and Kirchhoff, 2006; El-Desoky Mousa et al., 2013).”
Compositional Morphology for Word Representations and Language Modelling (2014, background): “Alexandrescu & Kirchhoff (2006) demonstrated how factorising the representations of context-words can help deal with out-of-vocabulary words, but they did not evaluate the effect of factorising output words and did not conduct an extrinsic evaluation.”
Alternative structures for character-level RNNs (2015, background): “We keep the most frequent words (between 3000 and 5000) and associate the other ones with the <UNK> token.”
Learning Character-level Representations for Part-of-Speech Tagging (2014, background): “(Alexandrescu & Kirchhoff, 2006) present a factored neural language model where each word is represented as a vector of features such as stems, morphological tags and cases.”
Unsupervised Morphology Induction Using Word Embeddings (2015, methodology): “Previous attempts at dealing with sub-word units and their compositionality have looked at explicitlyengineered features such as stems, cases, POS, etc., and used models such as factored NLMs (Alexandrescu and Kirchhoff, 2006) to obtain representations for unseen words, or compositional distribution”
Better Word Representations with Recursive Neural Networks for Morphology (2013, background): “This paper combines recursive neural networks (RNNs) and neural language models (NLMs) in a novel way to learn better word representations.”
Improving Word Sense Disambiguation in Neural Machine Translation with Sense Embeddings (2017, methodology): “In a ﬁrst approach, we compute sense embeddings and include the resulting sense labels into the NMT model as additional input features (Alexandrescu and Kirchhoff, 2006; Sennrich and Haddow, 2016).”
Factored recurrent neural network language model in TED lecture transcription (2012, background): “In addition, some studies [1, 2, 8, 12] introduced various syntactic features into their feed-forward NNLMs and discriminative language models.”
Adaptation Data Selection using Neural Language Models: Experiments in Machine Translation (2013, background): “Both types of neural LMs have seen many improvements recently, in terms of computational scalability (Le et al., 2011) and modeling power (Arisoy et al., 2012; Wu et al., 2012; Alexandrescu and Kirchhoff, 2006).”
Co-learning of Word Representations and Morpheme Representations (2014, background): “There are some other related works that consider morphological knowledge when learning the word embeddings, such as factored NLMs (Alexandrescu and Kirchhoff, 2006) and csmRNN (Luong et al.”
Statistical Language Models Based on Neural Networks (2012, background): “Integration of additional features into the NNLM framework (such as part of speech tags or morphology information) has been investigated in [19] [1].”
Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts (2014, background): “In terms of using intra-word information in neural network architectures for NLP tasks, Alexandrescu et al. (2006) present a factored neural language model where each word is represented as a vector of features such as stems, morphological tags and cases and a single embedding matrix is used to…”