An Analysis of Neural Language Modeling at Multiple Scales

By Stephen Merity, Nitish Shirish Keskar, Richard Socher. 2018

178 citations recorded by the source corpus.

Many of the leading approaches in language modeling introduce novel, complex and specialized architectures. We take existing state-of-the-art word level language models based on LSTMs and QRNNs and extend them to both larger vocabularies as well as character-level granularity. When properly tuned, LSTMs and QRNNs achieve state-of-the-art results on character-level (Penn Treebank, enwik8) and word-level (WikiText-103) datasets, respectively. Results are obtained in only 12 hours (WikiText-103) to 2 days (enwik8) using a single modern GPU.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing, Scaling & Learning Theory

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

In Scaling & Learning Theory

Citations in the corpus

Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “The “Transformer-XL (151M)” setting uses a standard parameter budget as previous work Merity et al. (2018), where we observe a similar effect when increasing the attention length during evaluation.”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “Merity et al. (2018) evaluates both character-based and word-based factorizations but does not directly compare them to each other.”
The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks (2018, methodology): “Next we apply our technique to one of the current state-ofthe-art world-level language models [35].”
Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information (2018, background): “Machine learning models for estimating the probabilities of potential next words (and hence, for predicting the next word) in a running text have seen enormous improvements in performance over the last few years (Merity et al., 2018).”
Improving Neural Language Modeling via Adversarial Training (2019, methodology): “On the large scale WikiText-103 (WT3) dataset, our method improves the Quasi-recurrent neural networks (QRNNs) (Merity et al., 2018b) baseline.”
Deep Anomaly Detection with Outlier Exposure (2018, methodology): “We use QRNN (Merity et al., 2018a;b) language models as baseline OOD detectors.”
Deep Equilibrium Models (2019, neutral): “2 AWD-QRNN [33] 159M 51M 33.”
Dynamic Evaluation of Transformer Language Models (2019, neutral): “…1.34 HM-LSTM (Chung et al., 2017) 35M 1.32 Recurrent highway networks (Zilly et al., 2017) 46M 1.27 FS-LSTM (Mujika et al., 2017) 47M 1.25 awd-LSTM (Merity et al., 2018) 47M 1.23 Transformer + aux losses (Al-Rfou et al., 2018) 235M 1.06 Multiplicative LSTM (Krause et al., 2016) 46M 1.24…”
Learning Dynamic Author Representations with Temporal Language Models (2019, background): “Research on these deep language models is very active [4]–[8], with applications in various text-related tasks such as speech recognition [9], image captioning [10], or text generation [11].”
On Training Recurrent Networks with Truncated Backpropagation Through time in Speech Recognition (2018, background): “Recurrent neural networks (RNNs) have been extensively used for speech and language processing, and are particularly successful at sequence prediction tasks, such as language modeling [1, 2, 3] and automatic speech recognition [4, 5, 6].”
Hardware-Guided Symbiotic Training for Compact, Accurate, yet Execution-Efficient LSTM (2019, methodology): “We use a dropout ratio of 0.2 for the hidden layers in the control gates, as in [18], 0.65 for input embedding layers, and 0.1 for input words, as in [37].”
The Implicit and Explicit Regularization Effects of Dropout (2020, methodology): “In this section, we show that the regularizers derived in Section 4 can replace dropout for LSTMs on language modeling tasks.”