Exploring the Limits of Language Modeling

By Rafal Józefowicz, Oriol Vinyals, Mike Schuster, Noam Shazeer, Yonghui Wu. 2016

1,188 citations recorded by the source corpus.

In this work we explore recent advances in Recurrent Neural Networks for large scale Language Modeling, a task central to language understanding. We extend current models to deal with two key challenges present in this task: corpora and vocabulary sizes, and complex, long term structure of language. We perform an exhaustive study on techniques such as character Convolutional Neural Networks or Long-Short Term Memory, on the One Billion Word Benchmark. Our best single model significantly improves state-of-the-art perplexity from 51.3 down to 30.0 (whilst reducing the number of parameters by a factor of 20), while an ensemble of models sets a new record by improving perplexity from 41.0 down to 23.7. We also release these models for the NLP and ML community to study and improve upon.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Factorization tricks for LSTM networks (2017, methodology): “As a baseline, we used BIGLSTM model without CNN inputs described by Jozefowicz et al. (2016).”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “Previous State-of-the-Art: The best previously published results (Jozefowicz et al., 2016) use models consisting of one or more stacked Long Short-Term Memory (LSTM) layers (Hochreiter & Schmidhuber, 1997; Gers et al., 2000).”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, background): “Among the RNN variants, LSTMs have been shown to provide state-of-the-art performance for several benchmark language modeling tasks (J´ozefowicz et al., 2016; Merity et al., 2018; Melis et al., 2018).”
SWAG: A Large-Scale Adversarial Dataset for Grounded Commonsense Inference (2018, methodology): “We tried more complicated language modeling architectures, such as from (Józefowicz et al., 2016), but ended up not seeing an improvement due to overfitting.”
Deep Contextualized Word Representations (2018, methodology): “To balance overall language model perplexity with model size and computational requirements for downstream tasks while maintaining a purely character-based input representation, we halved all embedding and hidden dimensions from the single best model CNN-BIG-LSTM in J ´ ozefowicz et al. (2016).”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “ollowed by two stacked LSTMs, each with 8192 hidden units and a 1024 dimensional projection layer. We use CNN-BIG-LSTM to refer to this language model in our results. In addition to CNN-BIG-LSTM from Jozefowicz et al. (2016),´ 1 we used the same corpus to train two additional language models with fe”
context2vec: Learning Generic Context Embedding with Bidirectional LSTM (2016, background): “…language models, as both train LSTM neural networks with the objective to predict target words based on their (short and long range) context, and both use techniques, such as negative sampling, to address large vocabulary computational challenges during training (Jozefowicz et al., 2016).”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, background): “…research argues that general methods that can leverage additional computation ultimately win out against methods that rely on human expertise [Sutton, 2019; Hestness et al., 2017; Shazeer et al., 2017; Jozefowicz et al., 2016; Mahajan et al., 2018; Shazeer et al., 2018, 2017; Huang et al., 2018b].”
SummAE: Zero-Shot Abstractive Text Summarization using Length-Agnostic Auto-Encoders (2019, background): “Recent progress in neural language models (Sutskever et al., 2014; Jozefowicz et al., 2016; Radford et al., 2019) has enabled models to generate near ﬂuent language that are not mere regurgitations of training data.”
The Woman Worked as a Babysitter: On Biases in Language Generation (2019, methodology): “We analyze OpenAI’s GPT-2 (small) language model (Radford et al., 2019) and Google’s language model trained on the One Bil-lion Word Benchmark (Jozefowicz et al., 2016).”
Defending Against Neural Fake News (2019, background): “Thus, while we are excited about recent progress in text generation (Józefowicz et al., 2016; Radford et al., 2018; 2019), we are also concerned with the inevitability of AI-generated ‘neural’ fake news.”
Generating Long Sequences with Sparse Transformers (2019, background): “Recently, neural autoregressive models have achieved impressive results in this domain, achieving state-of-the-art in modeling natural language (Jozefowicz et al., 2016) (Rad-ford et al., 2018) (Dai et al., 2018), raw audio (Van Den Oord et al., 2016) (Mehri et al., 2016), and images (Oord et al.,…”
Cross-lingual Language Model Pretraining (2019, background): “While recurrent neural networks obtain state-of-the-art performance on language modeling benchmarks (Mikolov et al., 2010; Jozefowicz et al., 2016), Transformer models are also very competitive (Dai et al., 2019).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “…al. (2016); Press & Wolf (2016); Merity et al. (2017); Gal & Ghahramani (2016), speeding up the Softmax computation (Morin & Bengio, 2005; Kuchaiev & Ginsburg, 2017; Grave et al., 2016a; Jozefowicz et al., 2016), and enriching the output distribution family (Yang et al., 2017; Kanai et al., 2018).”
A Simple Method for Commonsense Reasoning (2018, methodology): “Following [31], we employ importance sampling at the softmax layer with 8,192 negative samples for each mini-batch to signiﬁcantly speed up training.”