Scaling recurrent neural network language models

By Will Williams, Niranjani Prasad, D. Mrva, T. Ash, Tony A. Robinson. 2015

72 citations recorded by the source corpus.

This paper investigates the scaling properties of Recurrent Neural Network Language Models (RNNLMs). We discuss how to train very large RNNs on GPUs and address the questions of how RNNLMs scale with respect to model size, training-set size, computational costs and memory. Our analysis shows that despite being more costly to train, RNNLMs obtain much lower perplexities on standard benchmarks than n-gram models. We train the largest known RNNs and present relative word error rates gains of 18% on an ASR task. We also present the new lowest perplexities on the recently released billion word language modelling benchmark, 1 BLEU point gain on machine translation and a 17% relative hit rate gain in word prediction.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Exploring the Limits of Language Modeling (2016, background): “Much work has been done on small and large scale RNN-based LMs (Mikolov et al., 2010; Mikolov, 2012; Chelba et al., 2013; Zaremba et al., 2014; Williams et al., 2015; Ji et al., 2015a; Wang & Cho, 2015; Ji et al., 2015b).”
Batch normalized recurrent neural networks (2015, background): “Unfortunately, the training cost for large datasets and deep architectures of stacked RNNs can be prohibitively high, often times an order of magnitude greater than simpler models like n-grams [7].”
BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies (2015, methodology): “Following the experiments in Williams et al. (2015), we evaluate the performance of BlackOut on a vocabulary of 64K most frequent words.”
Generalizing and Hybridizing Count-based and Neural Language Models (2016, background): “On the other hand, these neural LMs also come at the cost of increased computational complexity at both training and test time, and even the largest reported neural LMs (Chen et al., 2015; Williams et al., 2015) are trained on a fraction of the data of their count-based counterparts (Brants et al.”
A Survey of the Usages of Deep Learning for Natural Language Processing (2020, unknown): “producing character-level outputs [55], [64]–[66].”
Frustratingly Short Attention Spans in Neural Language Modeling (2017, background): “Neural language models often work best in combination with traditional N -gram models (Mikolov et al., 2011; Chelba et al., 2013; Williams et al., 2015; Ji et al., 2016; Shazeer et al., 2015), since the former excel at generalization while the latter ensure memorization.”
Sparse Non-negative Matrix Language Modeling (2015, background): “Besides their excellent results, RNNs have also been shown to scale well with large amounts of data with regards to memory and accuracy (Williams et al., 2015).”