Learning Neural Trans-Dimensional Random Field Language Models with Noise-Contrastive Estimation

By Zhijian Ou. 2017

19 citations recorded by the source corpus.

Trans-dimensional random field language models (TRF LMs) where sentences are modeled as a collection of random fields, have shown close performance with LSTM LMs in speech recognition and are computationally more efficient in inference. However, the training efficiency of neural TRF LMs is not satisfactory, which limits the scalability of TRF LMs on large training corpus. In this paper, several techniques on both model formulation and parameter estimation are proposed to improve the training efficiency and the performance of neural TRF LMs. First, TRFs are reformulated in the form of exponential tilting of a reference distribution. Second, noise-contrastive estimation (NCE) is introduced to jointly estimate the model parameters and normalization constants. Third, we extend the neural TRF LMs by marrying the deep convolutional neural network (CNN) and the bidirectional LSTM into the potential function to extract the deep hierarchical features and bidirectionally sequential features. Utilizing all the above techniques enables the successful and efficient training of neural TRF LMs on a 40x larger training set with only 1/3 training time and further reduces the WER with relative reduction of 4.7% on top of a strong LSTM LM baseline.

Subfields: Artificial Intelligence, Natural Language Processing, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Luke S. Zettlemoyer
2. Graham Neubig
3. Mohit Bansal
4. Quoc V. Le
5. Jianfeng Gao
6. Mohit Iyyer
7. Myle Ott
8. Kyunghyun Cho

In Natural Language Processing

1. Graham Neubig
2. Luke S. Zettlemoyer
3. Mohit Iyyer
4. Mohit Bansal
5. Myle Ott
6. John Wieting
7. Jianfeng Gao
8. Kyunghyun Cho

In Representation & Self-Supervised Learning

1. Quoc V. Le
2. Minh-Thang Luong
3. Christopher D. Manning
4. Mohit Iyyer
5. Kevin Clark
6. John Wieting
7. Arthur Szlam
8. Marc'Aurelio Ranzato

In Signal Processing

1. Quoc V. Le
2. Anton Bakhtin
3. Myle Ott
4. Arthur Szlam
5. Yuntian Deng
6. Minh-Thang Luong
7. Christopher D. Manning
8. Kevin Clark

Citations in the corpus

• Residual Energy-Based Models for Text Generation (2020, methodology): “We investigated an EBM trained on the residual of a pretrained autoregressive language model (Wang & Ou, 2018b; Parshakova et al., 2019).”
• Pre-Training Transformers as Energy-Based Cloze Models (2020, methodology): “NCE commonly employs this idea to ensure the classification task is sufficiently challenging for the model (Gutmann and Hyv¨arinen, 2010; Wang and Ou, 2018).”
• Improved Training Of Neural Trans-Dimensional Random field Language Models with Dynamic Noise-Contrastive Estimation (2018, methodology): “Specifically, the neural TRF LMs [5, 6], whose potential function is defined by a neural network, have been shown to outperform the classical n-gram LMs significantly, and perform close to LSTM LMs but are computational more efficient in computing sentence probabilities.”
• RankGen: Improving Text Generation with Large Ranking Models (2022, background): “, 2019), speech recognition (Wang and Ou, 2018), open-ended text generation (Bakhtin et al.”
• Learning Energy-based Model with Flow-based Backbone by Neural Transport MCMC (2020, unknown): “Earlier works include [50, 56].”
• Residual Energy-Based Models for Text (2021, methodology): “EBMs have been used for sequence modeling (Rosenfeld et al., 2001; Wang et al., 2015, 2017; Wang and Ou, 2017, 2018a; Parshakova et al., 2019).”