Towards better integration of semantic predictors in statistical language modeling

By Noah B. Coccaro, Dan Jurafsky. 1998

We introduce a number of techniques designed to help integrate semantic knowledge with N-gram language models for automatic speech recognition. Our techniques allow us to integrate Latent Semantic Analysis (LSA), a word-similarity algorithm based on word co-occurrence information, with N-gram models. While LSA is good at predicting content words which are coherent with the rest of a text, it is a bad predictor of frequent words, has a low dynamic range, and is inaccurate when combined linearly with N-grams. We show that modifying the dynamic range, applying a per-word conﬁdence metric, and using geometric rather than linear combinations with N-grams produces a more robust language model which has a lower perplexity on a Wall Street Journal test-set than a baseline N-gram model.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Unbounded cache model for online language modeling with open vocabulary (2017, background): “Bellegarda [5] and Coccaro and Jurafsky [14] use latent semantic analysis to adapt their models to current context.”
Improving Neural Language Models with a Continuous Cache (2016, neutral): “Bellegarda (2000) and Coccaro & Jurafsky (1998) use latent semantic analysis to adapt their models to the current context.”
Integrating Topics and Syntax (2004, background): “[7]), each word would exhibit both short-range and long-range dep endencies.”
Exploiting latent semantic information in statistical language modeling (2000, background): “Recent work has sought to extend the word trigger concept by using a more comprehensive framework to handle the trigger pair selection [3]–[10], [12], [24], [34], [36].”
Composition in Distributional Models of Semantics (2010, methodology): “For example, they have been used to model human similarity judgements (McDonald, 2000), enhance n-gram language models with long range semantic information (Bellegarda, 2000; Coccaro and Jurafsky, 1998) and quantify the effect of semantic constraint on reading times (Pynte et al.”
Context dependent recurrent neural network language model (2012, result): “Similar in spir it to the cache based models are the latent semantic analysis (LSA) based approaches of [12, 13].”
Statistical language model adaptation: review and perspectives (2004, unknown): “Recent work has sought to extend the word trigger concept by using a more systematic framework to handle the trigger pair selection (Bellegarda, 1998a,b, 2000a,b; Coccaro and Jurafsky, 1998).”
Computational Approaches to Sentence Completion (2012, methodology): “To do this, we adopt the procedure of (Coccaro and Jurafsky, 1998), using linear interpolation between the n-gram and LSA probabilities:”
Polarity Inducing Latent Semantic Analysis (2012, background): “, 2009; Reisinger and Mooney, 2010) and for language modeling (Bellegarda, 2000; Coccaro and Jurafsky, 1998).”
Unsupervised Learning by Probabilistic Latent Semantic Analysis (2004, methodology): “We have thus followed the approach of Coccaro and Jurafsky (1998) to derive LSA probabilities.”
Vector-based Models of Semantic Composition (2008, methodology): “NLP tasks that could benefit from composition models include paraphrase identification and context-dependent language modeling (Coccaro and Jurafsky, 1998).”