Shrinking Exponential Language Models

By Stanley F. Chen. 2009

105 citations recorded by the source corpus.

In (Chen, 2009), we show that for a variety of language models belonging to the exponential family, the test set cross-entropy of a model can be accurately predicted from its training set cross-entropy and its parameter values. In this work, we show how this relationship can be used to motivate two heuristics for "shrinking" the size of a language model to improve its performance. We use the first heuristic to develop a novel class-based language model that outperforms a baseline word trigram model by 28% in perplexity and 1.9% absolute in speech recognition word-error rate on Wall Street Journal data. We use the second heuristic to motivate a regularized version of minimum discrimination information models and show that this method outperforms other techniques for domain adaptation.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Li Deng
2. Dong Yu
3. Yoshua Bengio
4. Jianfeng Gao
5. Geoffry Zweig
6. Quoc V. Le
7. Mohit Bansal
8. Richard Socher

Citations in the corpus

• Strategies for training large scale neural network language models (2011, result): “3% after rescoring with “model M” [21] (while in our experiments, rescoring with model M resulted in 12.”
• The Microsoft Research Sentence Completion Challenge (2011, methodology): “A state-of-the-art classbased maximum entropy n-gram model [7] was used to generate the alternates.”
• The IBM 2015 English conversational telephone speech recognition system (2015, methodology): “For LM rescoring, we used two types of LMs: model M, a class-based exponential model [25] and neural network LM (NNLM) [26, 27, 28, 29].”
• Trans-dimensional Random Fields for Language Modeling (2015, background): “Word classing has been shown to be useful in conditional ME models (Chen, 2009).”
• Statistical Language Models Based on Neural Networks (2012, methodology): “The baseline system for these experiments from IBM (RT04 Broadcast News speech recognition) has been recently used in the 2010 Summer Workshop at Johns Hopkins University [82].”
• Direct Acoustics-to-Word Models for English Conversational Speech Recognition (2017, background): “The above intuition led us to use GloVe embeddings [20] for initializing the final dense layer, similar to the technique proposed in [26] for augmenting neural network LMs with externally-trained GloVe embeddings.”
• Speed regularization and optimality in word classing (2013, methodology): “To further verify the usefulness of speed regularized classing, we repeated the experiments with an implementation of Model M [5, 6] using 4gm features.”
• Computational Approaches to Sentence Completion (2012, methodology): “We begin with n-gram models; first a classical n-gram backoff model (Chen and Goodman, 1999), and then a recently proposed class-based maximum-entropy n-gram model (Chen, 2009a; Chen, 2009b).”
• Learning Trans-Dimensional Random Fields with Applications to Language Modeling (2018, background): “n-gram LMs [6], neural network (NN) LMs [7], [8], [9] and conditional maximum entropy LMs [10], [11], [12].”
• A Challenge Set for Advancing Language Modeling (2012, methodology): “…techniques have been extensively refined, with state-of-the-art techniques based on smoothed N-gram counts (Kneser and Ney, 1995; Chen and Good-man, 1999), multi-layer perceptrons (Schwenk and Gauvain, 2002; Schwenk, 2007) and maximum-entropy models (Rosenfeld, 1997; Chen, 2009a; Chen, 2009b).”
• Recurrent neural network language model adaptation with curriculum learning (2015, methodology): “Cache-based statistical language models (Kuhn and de Mori, 1990; Jelinek et al., 1991) carry out dynamic adaptation during the process of recognition and fall under the category of model interpolation. These models represent one of the initial attempts to model long-distance dependency in language m”
• Context dependent recurrent neural network language model (2012, background): “Khudanpur and Wu [16] proposed the use of topic-features within a maximum-entropy framework, and in a voice-search application [17], long span context was used to enhance the maximum-entropy model of [18, 19] by creating features to indicate when a hypothesized word appeared in a user’s history.”