Improving Neural Language Modeling via Adversarial Training

By Dilin Wang, Chengyue Gong, Qiang Liu. 2019

127 citations recorded by the source corpus.

Recently, substantial progress has been made in language modeling by using deep neural networks. However, in practice, large scale neural language models have been shown to be prone to overfitting. In this paper, we present a simple yet highly effective adversarial training mechanism for regularizing neural language models. The idea is to introduce adversarial noise to the output embedding layer while training the models. We show that the optimal adversarial noise yields a simple closed-form solution, thus allowing us to develop a simple and time efficient algorithm. Theoretically, we show that our adversarial mechanism effectively encourages the diversity of the embedding vectors, helping to increase the robustness of models. Empirically, we show that our method improves on the single model state-of-the-art results for language modeling on Penn Treebank (PTB) and Wikitext-2, achieving test perplexity scores of 46.01 and 38.07, respectively. When applied to machine translation, our method improves over various transformer-based translation baselines in BLEU scores on the WMT14 English-German and IWSLT14 German-English tasks.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jianfeng Gao
2. Zhe Gan
3. Mohit Bansal
4. Jingjing Liu
5. Yu Cheng
6. Yejin Choi
7. Linjie Li
8. Noah A. Smith

In Natural Language Processing

1. Jianfeng Gao
2. Weizhu Chen
3. Xiaodong Liu
4. Pengcheng He
5. Zhe Gan
6. Jingjing Liu
7. Mohit Bansal
8. Yejin Choi

Citations in the corpus

• Adversarial Training for Large Neural Language Models (2020, background): “However, aside from Wang et al. (2019), there has not been any prior work on adversarial pre-training, and Wang et al. (2019) only applied adversarial training to generative language modeling using LSTM. ALUM is applicable to both pre-training and fine-tuning.”
• Large-Scale Adversarial Training for Vision-and-Language Representation Learning (2020, background): “There are also some parallel studies on applying AT to language modeling [67] and natural language understanding [45, 85, 24].”
• Measuring Association Between Labels and Free-Text Rationales (2020, methodology): “Method Following related work (Wang et al., 2019; Lakshmi Narayan et al., 2019; Liu et al., 2019b), we add zero-mean Gaussian noise N (0 , σ 2 ) to each input embedding in the I → OR encoder at inference time.”
• AutoML: A Survey of the State-of-the-Art (2019, neutral): “adversarial AWD-LSTM-Mos [149] and Transformer-XL [5].”
• A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly (2023, background): “[284] extended this approach to the continuous embedding space, facilitating more practical convergence, as followed by subsequent research [167, 338, 156].”
• A Survey on Data Augmentation for Text Classification (2021, unknown): “Wang et al. [127] and Liu et al. [128] developed methods for enhancing the pre-training of language models with adversarial training.”
• MaxUp: A Simple Way to Improve Generalization of Neural Network Training (2020, unknown): “We use the code provided by Wang et al. (2019) as our base-line 3 , which stacks a three-layer LSTM and implements a bag of regularization and optimization tricks for neural language modeling proposed by Merity et al. (2018), such as weight tying, word embedding drop and Averaged SGD.”
• Autoregressive Knowledge Distillation through Imitation Learning (2020, methodology): “Following standard practice (Bahdanau et al., 2016; Deng et al., 2018; Wang et al., 2019), we randomly sample 4% of this dataset as the validation set and let the remaining be the training set.”
• Weight Tying Biases Token Embeddings Towards the Output Space (2026, unknown): “…spaces compared to tied models, suggesting that untying may yield more uniform representations, a property shown to improve downstream performance (Gao et al., 2019; Etha-yarajh, 2019; Bi ´ s et al., 2021; Rajaee and Pilehvar, 2022; Wang et al., 2019, 2020; Stollenwerk and Stollenwerk, 2025).”