Language modeling with neural trans-dimensional random fields

By Zhijian Ou. 2017

17 citations recorded by the source corpus.

Trans-dimensional random field language models (TRF LMs) have recently been introduced, where sentences are modeled as a collection of random fields. The TRF approach has been shown to have the advantages of being computationally more efficient in inference than LSTM LMs with close performance and being able to flexibly integrate rich features. In this paper we propose neural TRFs, beyond of the previous discrete TRFs that only use linear potentials with discrete features. The idea is to use nonlinear potentials with continuous features, implemented by neural networks (NNs), in the TRF framework. Neural TRFs combine the advantages of both NNs and TRFs. The benefits of word embedding, nonlinear feature learning and larger context modeling are inherited from the use of NNs. At the same time, the strength of efficient inference by avoiding expensive softmax is preserved. A number of technical contributions, including employing deep convolutional neural networks (CNNs) to define the potentials and incorporating the joint stochastic approximation (JSA) strategy in the training algorithm, are developed in this work, which enable us to successfully train neural TRF LMs. Various LMs are evaluated in terms of speech recognition WERs by rescoring the 1000-best lists of WSJ'92 test data. The results show that neural TRF LMs not only improve over discrete TRF LMs, but also perform slightly better than LSTM LMs with only one fifth of parameters and 16x faster inference efficiency.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Residual Energy-Based Models for Text Generation (2020, methodology): “Energy Networks have been used for sequence modeling (Rosenfeld et al., 2001; Wang et al., 2015; 2017; Wang & Ou, 2017; 2018a; Parshakova et al., 2019).”
Learning Neural Trans-Dimensional Random Field Language Models with Noise-Contrastive Estimation (2017, methodology): “In contrast, a new trans-dimensional random field (TRF) LM [4, 5, 6] has been proposed in the undirected graphical modeling approach, where sentences are modeled as a collection of random fields and the joint probability is defined in terms of potential functions.”
Residual Energy-Based Models for Text (2021, methodology): “However, despite several promising eﬀorts (Rosenfeld et al., 2001; Wang et al., 2015, 2017; Wang and Ou, 2017, 2018a), they still have not been as successful in the text domain as locally-normalized auto-regressive models (Radford et al., 2019), which generate each word sequentially conditioned on…”
Improved Training Of Neural Trans-Dimensional Random field Language Models with Dynamic Noise-Contrastive Estimation (2018, methodology): “Specifically, the neural TRF LMs [5, 6], whose potential function is defined by a neural network, have been shown to outperform the classical n-gram LMs significantly, and perform close to LSTM LMs but are computational more efficient in computing sentence probabilities.”
Energy-Based Models for Text (2020, methodology): “However, despite several promising efforts (Rosenfeld et al., 2001; Wang et al., 2015, 2017; Wang and Ou, 2017, 2018a), they still have not been as successful in the text domain as locally-normalized auto-regressive models (Radford et al., 2019a,b), which can be trained efficiently via maximum…”