Maximum entropy techniques for exploiting syntactic, semantic and collocational dependencies in language modeling

By Sanjeev Khudanpur, Jun Wu. 2000

108 citations recorded by the source corpus.

A new statistical language model is presented which combines collocational dependencies with two important sources of long-range statistical dependence: the syntactic structure and the topic of a sentence. These dependencies or constraints are integrated using the maximum entropy technique. Substantial improvements are demonstrated over a trigram model in both perplexity and speech recognition accuracy on the Switchboard task. A detailed analysis of the performance of this language model is provided in order to characterize the manner in which it performs better than a standard N -gram model. It is shown that topic dependencies are most useful in predicting words which are semantically related by the subject matter of the conversation. Syntactic dependencies on the other hand are found to be most helpful in positions where the best predictors of the following word are not within N -gram range due to an intervening phrase or clause. It is also shown that these two methods individually enhance an N -gram model in complementary ways and the overall improvement from their combination is nearly additive.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Natural Language Processing

Citations in the corpus

Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “Similarly, topic features have been used with either maximum entropy models [33] or recurrent networks [46, 53].”
Improving Neural Language Models with a Continuous Cache (2016, methodology): “Similarly, topic features have been used with either maximum entropy models (Khudanpur & Wu, 2000) or recurrent networks (Mikolov & Zweig, 2012; Wang & Cho, 2015).”
Discriminative Language Modeling with Conditional Random Fields and the Perceptron Algorithm (2004, methodology): “While we focus on n-gram models, we stress that our methods are applicable to more general language modeling features – for example, syntactic features, as explored in, e.g., Khudanpur and Wu (2000).”
Bootstrapping parsers via syntactic projection across parallel texts (2005, methodology): “…of the parallel text is analyzed by Collins’s (1997) Model 2 parser (trained on the Wall Street Journal portion of the Penn Treebank) and then converted to a dependency representation based on a standard head-table approach (Magerman 1994), using an algorithm similar that of Xia and Palmer (2001).”
A Challenge Set for Advancing Language Modeling (2012, methodology): “…of N-gram modeling techniques, there has been an important stream of research focused on the incorporation of syntactic and semantic information (Chelba and Jelinek, 1998; Chelba and Jelinek, 2000; Rosenfeld et al., 2001; Yamada and Knight, 2001; Khudanpur and Wu, 2000; Wu and Khudanpur, 1999).”
Recurrent neural network based language model personalization by social network crowdsourcing (2013, background): “However, the lack of natural strategy to model long-range dependencies [19, 20, 21], and the potentially error-prone back-off behaviors [22] seem to limit the prediction power of N-gram models.”
Exploiting prosodic breaks in language modeling with random forests (2008, methodology): “So by asking the RFLM not to use one of the variables in the history, we can find out how valuable that feature is. This kind of feature engineering was also used in maximum entropy LMs [15, 16 ].”
Context dependent recurrent neural network language model (2012, background): “Khudanpur and Wu [16] proposed the use of topic-features within a maximum-entropy framework, and in a voice-search application [17], long span context was used to enhance the maximum-entropy model of [18, 19] by creating features to indicate when a hypothesized word appeared in a user’s history.”
Trans-dimensional Random Fields for Language Modeling (2015, background): “The conditional ME models (Khudanpur and Wu, 2000; Roark et al., 2004) suffer from the expensive computation of local normalization factors.”