On training bi-directional neural network language model with noise contrastive estimation

By Tianxing He, Yu Zhang, Jasha Droppo, Kai Yu. 2016

18 citations recorded by the source corpus.

Although uni-directional recurrent neural network language model(RNNLM) has been very successful, it's hard to train a bi-directional RNNLM properly due to the generative nature of language model. In this work, we propose to train bi-directional RNNLM with noise contrastive estimation(NCE), since the properities of NCE training will help the model to acheieve sentence-level normalization. Experiments are conducted on two hand-crafted tasks on the PTB data set: a rescore task and a sanity test. Although(regretfully), the model trained by NCE did not out-perform the baseline uni-directional NNLM, it is shown that NCE-trained bi-directional NNLM behaves well in the sanity test and outperformed the one trained by conventional maximum likelihood training on the rescore task.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Erik Nijkamp
2. Ying Nian Wu
3. Song-Chun Zhu
4. Ruiqi Gao
5. Zhijian Ou
6. Quoc V. Le
7. Andrew M. Dai
8. Myle Ott

In Large Language Models

1. Zhijian Ou
2. Diederik P. Kingma
3. Kyomin Jung
4. Erik Nijkamp
5. Joo-Ah Shin
6. Andrew M. Dai
7. Yoonhyung Lee
8. Arthur Szlam

In Natural Language Processing

1. Zhijian Ou
2. Diederik P. Kingma
3. Kyomin Jung
4. Erik Nijkamp
5. Joo-Ah Shin
6. Andrew M. Dai
7. Yoonhyung Lee
8. Ying Nian Wu

Citations in the corpus

• Flow Contrastive Estimation of Energy-Based Models (2019, background): “NCE and it variants has gained popularity in natural language processing (NLP) [23, 57, 2, 4].”
• Effective Sentence Scoring Method Using BERT for Speech Recognition (2019, background): “In natural language processing (NLP), many bidirectional language models have been studied He et al. (2016); Peters et al. (2018a); Devlin et al. (2018).”
• Joint Energy-based Model Training for Better Calibrated Natural Language Understanding Models (2021, unknown): “It has also been used in an attempt to train a sentence-level bi-directional LM (He et al., 2016).”
• Improved Training Of Neural Trans-Dimensional Random field Language Models with Dynamic Noise-Contrastive Estimation (2018, result): “Compared with the bidirectional LSTM LMs in [22, 23], this neural TRF LM provides a theoretical-solid framework to incorporate the bidirectional LSTM features.”