Factored Language Models and Generalized Parallel Backoff

By J. Bilmes, Katrin Kirchhoff. North American Chapter of the Association for Computational Linguistics, 2003

332 citations recorded by the source corpus.

We introduce factored language models (FLMs) and generalized parallel backoff (GPB). An FLM represents words as bundles of features (e.g., morphological classes, stems, data-driven clusters, etc.), and induces a probability model covering sequences of bundles rather than just words. GPB extends standard backoff to general conditional probability tables where variables might be heterogeneous types, where no obvious natural (temporal) backoff order exists, and where multiple dynamic backoff strategies are allowed. These methodologies were implemented during the JHU 2002 workshop as extensions to the SRI language modeling toolkit. This paper provides initial perplexity results on both CallHome Arabic and on Penn Treebank Wall Street Journal articles. Significantly, FLMs with GPB can produce bigrams with significantly lower perplexity, sometimes lower than highly-optimized baseline trigrams. In a multi-pass speech recognition context, where bigrams are used to create first-pass bigram lattices or N-best lists, these results are highly relevant.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c20b4068be640ebaffbc56382c3e4e0bcf62664e

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Alternative structures for character-level RNNs (2015, background): “We keep the most frequent words (between 3000 and 5000) and associate the other ones with the <UNK> token.”
Hierarchical Bayesian Language Models for Conversational Speech Recognition (2010, neutral): “Approaches which aim to improve on maximum-likelihood -gram models of word sequences include neural network-based models [8], latent variable models [9], and a Bayesian framework [10]–[12].”
Linguistically-augmented perplexity-based data selection for language models (2015, methodology): “While we have used one type of linguistic information in each model, another possibility is to combine different types of linguistic knowledge in a single LM, following for example the factored LM approach (Bilmes and Kirchhoff, 2003).”
Linguistic I Ssues in L Anguage Technology Lilt on Achieving and Evaluating Language-independence in Nlp on Achieving and Evaluating Language-independence in Nlp (background): “This is eectively what Bilmes and Kirchho ( 2003 ) did in generalizing n - gram language models to factored language models .”
Exploiting prosodic breaks in language modeling with random forests (2008, background): “, [22, 23]) or tried to find the optimal back-off path or combination of paths [24, 25].”
Factored Neural Language Models (2006, methodology): “A more powerful backoff strategy is used in factored language models (FLMs) (Bilmes and Kirchhoff, 2003), which view a word as a vector of word features or (cid:147)factors(cid:148): w = h f 1 , f 2 , . . . , f k and predict a word jointly from previous words and their factors: A generalized…”
Factored recurrent neural network language model in TED lecture transcription (2012, background): “Therefore, it can address the optimization problem well in factored n-gram LM [4, 7].”
Encoding linear models as weighted finite-state transducers (2014, background): “Similar issues arise for other complex model topologies, e.g., factored language models [8].”
Phrase-Based Statistical Language Generation Using Graphical Models and Active Learning (2010, methodology): “2: P (R|S∗) = L∏ t=1 P (rt|rt−1, ht, lt−1, lt, lt+1, s∗t−1, s ∗ t , s ∗ t+1) (8) Conditional probability distributions are represented as factored language models smoothed using Witten-Bell interpolated backoff smoothing (Bilmes and Kirchhoff, 2003), according to the backoff graphs in Fig.”
Continuous Space Translation Models with Neural Networks (2012, methodology): “Another is to use factored language models , introduced in (Bilmes and Kirchhoff, 2003), then adapted for translation models in (Koehn and Hoang, 2007; Crego and Yvon, 2010).”
Moses: Open Source Toolkit for Statistical Machine Translation (2007, background): “Moses is also able to integrate factored language models, such as those described in (Bilmes and Kirchhoff 2003) and (Axelrod 2006).”
Using Prosodic Features in Language Models for Meetings (2007, methodology): “Combining with interpolation or generalized parallel backoff (GPB) [7] strategies, multiple backoff paths may be used simultaneously.”
Compositional Morphology for Word Representations and Language Modelling (2014, methodology): “The primary outcomes are that (i) our morphology-guided CSLMs improve intrinsic language model performance when compared to base-line CSLMs and n -gram MKN models; (ii) word and morpheme representations learnt in the process compare favourably in terms of a word similarity task to a recent more comp”
English-to-Czech Factored Machine Translation (2007, background): “We use either one LM per output factor, varying the corpus size (NC or mix), or two LMs per output factors with weights trained independently in the MERT proce-236 dure (weighted).”
Linguistically Naïve != Language Independent: Why NLP Needs Linguistic Typology (2009, background): “This is effectively what Bilmes and Kirchhoff (2003) did in generalizing n -gram language models to factored language models.”
Character-Aware Neural Language Models (2015, unknown): “ 2007), and convolutional (Wang et al. 2015) networks. In order to address the rare word problem, Alexandrescu and Krichhoff (2006)—building on analogous work on count-based n-gram language models by Bilmes and Kirchhoff (2003)—represent a word as a set of shared factor embeddings. Their Factored Ne”
Square One Bias in NLP: Towards a Multi-Dimensional Exploration of the Research Manifold (2022, methodology): “Such models were nevertheless more commonly used than more linguistically informed alternatives such as factored language models (Bilmes and Kirchhoff, 2003) that represent words as sets of features.”