Trans-dimensional Random Fields for Language Modeling

By Zhijian Ou, Zhiqiang Tan. 2015

17 citations recorded by the source corpus.

Language modeling (LM) involves determining the joint probability of words in a sentence. The conditional approach is dominant, representing the joint probability in terms of conditionals. Examples include n-gram LMs and neural network LMs. An alternative approach, called the random field (RF) approach, is used in whole-sentence maximum entropy (WSME) LMs. Although the RF approach has potential benefits, the empirical results of previous WSME models are not satisfactory. In this paper, we revisit the RF approach for language modeling, with a number of innovations. We propose a trans-dimensional RF (TDRF) model and develop a training algorithm using joint stochastic approximation and trans-dimensional mixture sampling. We perform speech recognition experiments on Wall Street Journal data, and find that our TDRF models lead to performances as good as the recurrent neural network LMs but are computationally more efficient in computing sentence probability.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

Residual Energy-Based Models for Text Generation (2020, methodology): “Energy Networks have been used for sequence modeling (Rosenfeld et al., 2001; Wang et al., 2015; 2017; Wang & Ou, 2017; 2018a; Parshakova et al., 2019).”
Improved Training Of Neural Trans-Dimensional Random field Language Models with Dynamic Noise-Contrastive Estimation (2018, background): “TRF LMs are first proposed in [3] and further presented in [4] with thorough theoretical analysis and systematic evaluation.”
Language modeling with neural trans-dimensional random fields (2017, methodology): “The undirected graphical modeling approach, has few priori work, except [4, 5, 9].”
Pre-Training Transformers as Energy-Based Cloze Models (2020, methodology): “While EBMs have previously been applied to left-to-right (Wang et al., 2015) or globally normalized (Rosenfeld et al., 2001; Deng et al., 2020) text generation, they have not previously been applied to cloze models or for pre-training NLP models.”
Learning Trans-Dimensional Random Fields with Applications to Language Modeling (2018, methodology): “To compare the convergence of SA training using online estimated variances in [21] (green dash line) and using empirical variances in this paper (red solid line), and Adam method [22] (blue dot line), we plot an example of the scaled negative log-likelihoods on PTB training set along the iterations.”
Residual Energy-Based Models for Text (2021, methodology): “However, despite several promising eﬀorts (Rosenfeld et al., 2001; Wang et al., 2015, 2017; Wang and Ou, 2017, 2018a), they still have not been as successful in the text domain as locally-normalized auto-regressive models (Radford et al., 2019), which generate each word sequentially conditioned on…”
Learning Neural Trans-Dimensional Random Field Language Models with Noise-Contrastive Estimation (2017, background): “In contrast, a new trans-dimensional random field (TRF) LM [4, 5, 6] has been proposed in the undirected graphical modeling approach, where sentences are modeled as a collection of random fields and the joint probability is defined in terms of potential functions.”