Statistical language modeling using the CMU-cambridge toolkit

By Philip Clarkson, R. Rosenfeld. 1997

The CMU Statistical Language Modeling toolkit was released in 1994 in order to facilitate the construction and testing of bigram and trigram language models. It is currently in use in over 40 academic, government and industrial laboratories in over 12 countries. This paper presents a new version of the toolkit. We outline the conventional language modeling technology, as implemented in the toolkit, and describe the extra e(cid:14)ciency and functionality that the new toolkit provides as compared to previous software for this task. Finally, we give an example of the use of the toolkit in constructing and testing a simple language model.

Subfields: Natural Language Processing, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Speech & Audio

Citations in the corpus

Statistical Phrase-Based Translation (2003, methodology): “For all our experiments we use the same training data, trigram language model [Seymore and Rosenfeld, 1997], and a specialized decoder.”
An Empirical Study of Smoothing Techniques for Language Modeling (1996, background): “It should be noted that there exist language modeling toolkits (Rosenfeld, 1995; Clarkson and Rosenfeld, 1997) which can be used to build smoothed n-gram models using a variety of smoothing algorithms, including Katz smoothing, Jelinek-Mercer smoothing, absolute discounting, and Witten-Bell smoothin”
Stochastic Language Generation for Spoken Dialogue Systems (2000, methodology): “The CMU corpus was manually tagged, and back-off trigram models built (using Clarkson and Rosenfeld, 1997).”
The Smooth Signal Redundancy Hypothesis: A Functional Explanation for Relationships between Redundancy, Prosodic Prominence, and Duration in Spontaneous Speech (2004, background): “3 Good Turing and backoff are means of dealing with data sparsity and missing contexts when calculating trigram statistics (see Clarkson & Rosenfeld, 1997 for details).”
Natural Language Generation as Incremental Planning Under Uncertainty: Adaptive Information Presentation for Statistical Dialogue Systems (2014, methodology): “In order to account for data sparsity, we apply different discounting (“smoothing”) techniques including back-off, using the CMU Statistical Language Modelling toolkit [57], see Table III.”
SRILM - an extensible language modeling toolkit (2002, methodology): “Class N-grams are approximated by expanding class labels into their members (which can contain multiword strings) and then computing the marginal probabilities of word N-gram strings.”
A Phrase-Based,Joint Probability Model for Statistical Machine Translation (2002, methodology): “The probability p(E) is computed using a simple trigram language model that was trained using the CMU Language Modeling Toolkit (Clarkson and Rosenfeld, 1997).”
Rumor has it: Identifying Misinformation in Microblogs (2011, methodology): “− To build these language models, we use the CMU Language Modeling toolkit (Clarkson and Rosenfeld, 1997).”
Comparative Experiments on Sentiment Classification for Online Product Reviews (2006, methodology): “We employ the CMU-Cambridge Language Modeling Toolkit (Clarkson & Rosenfeld 1997) for the training and testing of language models.”
Recent Development of Open-Source Speech Recognition Engine Julius (2009, unknown): “Standard file formats are adopted to cope with other standard toolkits such as HTK (HMM Toolkit)[1], CMU-Cam SLM toolkit[2] and SRILM[3].”
Sphinx-4: a flexible open source framework for speech recognition (2004, methodology): “• LargeTrigramModel: provides support for true N-Gram models generated by the CM U-Cambridge Statistical Language Modeling Toolkit [22].”
Connectionist speech recognition of Broadcast News (2002, methodology): “The models were trained using version 2 of the CMU-Cambridge Statistical Language Model Toolkit (Clarkson and Rosenfeld 1997).”