One billion word benchmark for measuring progress in statistical language modeling

By Ciprian Chelba, Tomas Mikolov, Mike Schuster, Qi Ge, Thorsten Brants, Philipp Koehn, Tony A. Robinson. 2013

1,206 citations recorded by the source corpus.

We propose a new benchmark corpus to be used for measuring progress in statistical language modeling. With almost one billion words of training data, we hope this benchmark will be useful to quickly evaluate novel language modeling techniques, and to compare their contribution when combined with other advanced techniques. We show performance of several well-known types of language models, with the best results achieved with a recurrent neural network based language model. The baseline unpruned KneserNey 5-gram model achieves perplexity 67.6. A combination of techniques leads to 35% reduction in perplexity, or 10% reduction in cross-entropy (bits), over that baseline. The benchmark is available as a code.google.com project; besides the scripts needed to rebuild the training/held-out data, it also makes available log-probability values for each word in each of ten held-out data sets, for each of the baseline n-gram models.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019, methodology): “It is critical to use a document-level corpus rather than a shuffled sentence-level corpus such as the Billion Word Benchmark (Chelba et al., 2013) in order to extract long contiguous sequences.”
Factorization tricks for LSTM networks (2017, methodology): “We test F-LSTM and G-LSTM architectures on the task of language modeling using One Billion Word Benchmark (Chelba et al., 2013).”
Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer (2017, methodology): “1 , as described in Section 4 and Appendix A. Results: We evaluate our model using perplexity on the holdout dataset, used by (Chelba et al., 2013; Jozefowicz et al., 2016).”
Exploring the Limits of Language Modeling (2016, methodology): “Much work has been done on small and large scale RNN-based LMs (Mikolov, 2012; Chelba et al., 2013; Zaremba et al., 2014; Williams et al., 2015; Ji et al., 2015a; Wang & Cho, 2015; Ji et al., 2015b).”
Semi-Supervised Sequence Modeling with Cross-View Training (2018, methodology): “We use the 1 Billion Word Language Model Benchmark (Chelba et al., 2014) as a pool of unlabeled sentences for semi-supervised learning.”
Dissecting Contextual Word Embeddings: Architecture and Representation (2018, methodology): “Billion Word Benchmark (Chelba et al., 2014) using a sampled softmax with 8192 negative samples per batch.”
GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding (2018, methodology): “ELMo uses a pair of two-layer neural language models (one forward, one backward) trained on the One Billion Word Benchmark (Chelba et al., 2013).”
Deep Contextualized Word Representations (2018, methodology): “After training for 10 epochs on the 1B Word Benchmark (Chelba et al., 2014), the average forward and backward perplexities is 39.7, compared to 30.0 for the forward CNN-BIG-LSTM .”
Semi-supervised sequence tagging with bidirectional language models (2017, methodology): “The primary bidirectional LMs we used in this study were trained on the 1B Word Benchmark (Chelba et al., 2014), a publicly available benchmark for large-scale language modeling.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, background): “We apply Transformer-XL to a variety of datasets on both word-level and character-level language modeling to have a comparison with state-of-the-art systems, including WikiText-103 (Merity et al., 2016), enwiki8 (LLC, 2009), text8 (LLC, 2009), One Billion Word (Chelba et al., 2013), and Penn Treeban”
Deep Learning Scaling is Predictable, Empirically (2017, methodology): “We train the models on shards ranging from 0.1% up to 40% of the Billion Word Dataset.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “Outside Tasks We train language models on two datasets: WikiText-103 ( WT ; Merity et al., 2017) and Billion Word Language Model Benchmark ( BWB ; Chelba et al., 2013).”
Measuring the Effects of Data Parallelism on Neural Network Training (2018, background): “Chen et al. (2018) provide experimental evidence of a problemdependent critical batch size after which a b-fold benefit is no longer achieved for plain mini-batch SGD.”
Adaptive Input Representations for Neural Language Modeling (2018, background): “BILLION WORD contains 768M word tokens and has a vocabulary of about 800K word types, which corresponds to words with more than 3 occurrences in the training set (Chelba et al., 2013).”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “• To investigate whether the effectiveness of MoS can be extended to even larger datasets, we conduct an additional language modeling experiment on the 1B Word dataset (Chelba et al., 2013).”
One Epoch Is All You Need (2019, background): “The dataset used is 1 Billion Word Language Model Benchmark (LM1B).”