Regularizing and Optimizing LSTM Language Models

By Stephen Merity, Nitish Shirish Keskar, Richard Socher. 2017

1,163 citations recorded by the source corpus.

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

Subfields: Artificial Intelligence, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Optimization & Training

Citations in the corpus

Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “Among the RNN variants, LSTMs have been shown to provide state-of-the-art performance for several benchmark language modeling tasks (J´ozefowicz et al., 2016; Merity et al., 2018; Melis et al., 2018).”
Deep Contextualized Word Representations (2018, background): “(Józefowicz et al., 2016; Melis et al., 2017; Merity et al., 2017) compute a context-independent token representation xLM k (via token embeddings or a CNN over characters) then pass it through L layers of forward LSTMs.”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “We use the AWD-LSTM language model (Merity et al., 2017a) with an embedding size of 400 , 3 layers, 1150 hidden activations per layer, and a BPTT batch size of 70 .”
A Constructive Prediction of the Generalization Error Across Scales (2019, methodology): “For language modeling, we train AWD-LSTM (Merity et al., 2018) and Transformer-XL models (Dai et al.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “…al., 2016; Al-Rfou et al., 2018), improving regularization and optimization algorithms Zaremba et al. (2014); Inan et al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al.,…”
CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, neutral): “25 as in (Merity et al., 2017).”
Fine-tuned Language Models for Text Classification (2018, methodology): “In practice, we use variable length backpropagation sequences (Merity et al., 2017a).”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “Moreover, as deep learning models were tuned by extensive hyper-parameter search, increasing the dimension d beyond several hundred is not helpful3 (Merity et al., 2017; Melis et al., 2017; Krause et al., 2017).”
How Can We Know What Language Models Know? (2019, background): “Recent years have seen the primary role of language models (LMs) transition from generating or evaluating the fluency of natural text (Mikolov and Zweig, 2012; Merity et al., 2018; Melis et al., 2018; Gamon et al., 2005) to being a powerful tool for text understanding.”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, methodology): “We train this model on WikiText-103 dataset [34], a 500MB cleaned subset of English Wikipedia.”
High-Performance Large-Scale Image Recognition Without Normalization (2021, background): “Gradient clipping is often used in language modeling to stabilize training (Merity et al., 2018), and recent work shows that it allows training with larger learning rates compared to gradient descent, accelerating convergence (Zhang et al., 2020).”
On Network Design Spaces for Visual Recognition (2019, background): “For example, in natural language processing, recent large-scale studies [25, 26] demonstrate that when design spaces are well explored, the original LSTM [10] can outperform more recent models on language modeling benchmarks.”
Identifying and Reducing Gender Bias in Word-Level Language Models (2019, methodology): “We use a three-layer LSTM word-level language model (AWD-LSTM; Merity et al., 2018) with 1150 hidden units implemented in PyTorch.”
Learning to Write with Cooperative Discriminators (2018, methodology): “C ACHE LM As another LM baseline we include a continuous cache language model (Grave et al., 2017) as implemented by Merity et al. (2018), which recently obtained state-of-the-art perplexity on the Penn Treebank corpus (Marcus et al., 1993).”
ERNIE: Enhanced Language Representation with Informative Entities (2019, background): “Howard and Ruder (2018) present AWD-LSTM (Merity et al., 2017) to build a universal language model (ULMFiT).”
Born Again Neural Networks (2018, methodology): “Although we did not use the state of the art bag of tricks (Merity et al., 2017) for training LSTMs, nor the recently proposed improvements on knowledge distillation for sequence models (Kim & Rush, 2016), we found reasonable decreases in perplexity on both validation and testing set for our benchma”
DARTS: Differentiable Architecture Search (2018, methodology): “Other training settings are identical to those in Merity et al. (2017); Yang et al. (2017).”
Decentralized Deep Learning with Arbitrary Communication Compression (2019, methodology): “For the language modeling task, we borrowed and adapted the general experimental setup of (Merity et al., 2017), where we use a three-layer LSTM with hidden dimension of size 650 .”