Whole-sentence exponential language models: a vehicle for linguistic-statistical integration

By R. Rosenfeld, Stanley F. Chen, Xiaojin Zhu. 2001

110 citations recorded by the source corpus.

We introduce an exponential language model which models a whole sentence or utterance as a single unit. By avoiding the chain rule, the model treats each sentence as a “bag of features", where features are arbitrary computable properties of the sentence. The new model is computationally more efficient, and more naturally suited to modeling global sentential phenomena, than the conditional exponential (e.g. maximum entropy) models proposed to date. Using the model is straightforward. Training the model requires sampling from an exponential distribution. We describe the challenge of applying Monte Carlo Markov Chain and other sampling techniques to natural language, and discuss smoothing and step-size selection. We then present a novel procedure for feature selection, which exploits discrepancies between the existing model and the training corpus. We demonstrate our ideas by constructing and analysing competitive models in the Switchboard and Broadcast News domains, incorporating lexical and syntactic information.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Residual Energy-Based Models for Text Generation (2020, methodology): “Energy Networks have been used for sequence modeling (Rosenfeld et al., 2001; Wang et al., 2015; 2017; Wang & Ou, 2017; 2018a; Parshakova et al., 2019).”
A bit of progress in language modeling (2001, neutral): “A recent variation of the maximum entropy approach is the whole sentence maximum entropy approach (Rosenfeld et al., 2001).”
A Distributional Approach to Controlled Text Generation (2020, background): “The early work on ”Whole sentence exponential models” by (Rosenfeld et al., 2001) — which only came to our attention when preparing the final version of this paper — can be considered as a form of EBM over texts.”
Learning Trans-Dimensional Random Fields with Applications to Language Modeling (2018, methodology): “Although WSME models have the potential benefits of being able to naturally express sentence-level phenomena and integrate features from a variety of knowledge sources, their performance results ever reported are not satisfactory [5], [13], [14] (see Section 2.”
Trans-dimensional Random Fields for Language Modeling (2015, methodology): “After incorporating lexical and syntactic information, a mere relative improvement of 1% and 0.4% respectively in perplexity and in WER is reported for the resulting WSEM (Rosenfeld et al., 2001).”
Statistical language model adaptation: review and perspectives (2004, methodology): “This is the case of the wholesentence exponential model recently proposed in (Rosenfeld, 1997; Rosenfeld et al., 2001 ).”
Adaptive Importance Sampling to Accelerate Training of a Neural Probabilistic Language Model (2008, methodology): “The idea of applying sampling techniques to speedup language models, e.g., of the exponential family is not new. See, for example, [ 19 ].”
Sparse Non-negative Matrix Language Modeling (2015, background): “Estimating full sentence language models (Rosenfeld et al., 2001) is computationally hard if one seeks a properly normalized probability model1 over strings of words of finite length in V∗.”
Energy-Based Models for Text (2020, methodology): “EBMs have been used for sequence modeling (Rosenfeld et al., 2001; Wang et al., 2015, 2017; Wang and Ou, 2017, 2018a; Parshakova et al., 2019).”
Bidirectional Language Model for Handwriting Recognition (2012, background): “While such models have been proposed for speech recognition in a sophisticated way [14], our approach does not increase the language modeling and hence the computational complexity.”
Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC (2020, unknown): “Earlier works include [50, 56].”
Pre-Training Transformers as Energy-Based Cloze Models (2020, methodology): “While EBMs have previously been applied to left-to-right (Wang et al., 2015) or globally normalized (Rosenfeld et al., 2001; Deng et al., 2020) text generation, they have not previously been applied to cloze models or for pre-training NLP models.”
Residual Energy-Based Models for Text (2021, methodology): “However, despite several promising eﬀorts (Rosenfeld et al., 2001; Wang et al., 2015, 2017; Wang and Ou, 2017, 2018a), they still have not been as successful in the text domain as locally-normalized auto-regressive models (Radford et al., 2019), which generate each word sequentially conditioned on…”
A Challenge Set for Advancing Language Modeling (2012, background): “…of N-gram modeling techniques, there has been an important stream of research focused on the incorporation of syntactic and semantic information (Chelba and Jelinek, 1998; Chelba and Jelinek, 2000; Rosenfeld et al., 2001; Yamada and Knight, 2001; Khudanpur and Wu, 2000; Wu and Khudanpur, 1999).”