N-gram Counts and Language Models from the Common Crawl

By Christian Buck, Kenneth Heafield, Bas van Ooyen. 2014

194 citations recorded by the source corpus.

We contribute 5-gram counts and language models trained on the Common Crawl corpus, a collection over 9 billion web pages. This release improves upon the Google n-gram counts in two key ways: the inclusion of low-count entries and deduplication to reduce boilerplate. By preserving singletons, we were able to use Kneser-Ney smoothing to build large language models. This paper describes how the corpus was processed with emphasis on the problems that arise in working with data at this scale. Our unpruned Kneser-Ney English $5$-gram language model, built on 975 billion deduplicated tokens, contains over 500 billion unique n-grams. We show gains of 0.5-1.4 BLEU by using large language models to translate into various languages.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation (2016, neutral): “Using a human side-by-side evaluation on a set of isolated simple sentences, it reduces translation errors by an average of 60% compared to Google’s phrase-based production system.”
Deep Recurrent Models with Fast-Forward Connections for Neural Machine Translation (2016, methodology): “We find our best score outperforms the previous best score by nearly 3 points.”
Effective Approaches to Attention-based Neural Machine Translation (2015, methodology): “Winning WMT’14 system – phrase-based + large LM (Buck et al., 2014) 20.”
Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2019, methodology): “Common Crawl has previously been used as a source of text data for NLP, for example to train an n-gram language model [Buck et al., 2014], for commonsense reasoning [Trinh and Le, 2018], for mining parallel texts for machine translation [Smith et al.”
LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “This allowed to train a 5-gram model on 975 billions tokens from CommonCrawl, resulting in a model with 500 billions n-grams (Buck et al., 2014).”
Edinburgh’s Phrase-based Machine Translation Systems for WMT-14 (2014, background): “A full description of the pipeline, including a public data release, appears in Buck et al. (2014).”
Learning Word Vectors for 157 Languages (2018, methodology): “Removing duplicates is important for the crawl data, since it contains large amounts of boilerplate, as previously noted by Buck et al. (2014).”
CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Data (2019, methodology): “Common Crawl has been used in the context of language modeling to evaluate n-gram statistics (Buck et al., 2014).”
Unbounded cache model for online language modeling with open vocabulary (2017, methodology): “Thus, we always interpolate the probability distributions of the various models with the uniform distribution over the full vocabulary: This is a standard technique, which was previously used to compare language models trained on datasets with different vocabularies [9].”
Symbolic Knowledge Distillation: from General Language Models to Commonsense Models (2021, methodology): “Past automatic data mainly use extractive approaches, e.g. syntactic parsing (Zhang et al., 2020a) or pattern matching (Li et al., 2020) from unstructured text (Lehmann et al., 2015; Buck et al., 2014).”
On Using Very Large Target Vocabulary for Neural Machine Translation (2014, methodology): “On the English → German task, the best performance of 21.59 BLEU by our model is higher than that of the previous state of the art (20.67) reported in (Buck et al., 2014).”
Character-Aware Neural Language Models (2015, background): “While simplicity of training such models has made it possible to train them on corpora with over a trillion word tokens (Buck et al. 2014), they treat each word as a completely independent unit of information, and hence probabilities of rare n-grams can be poorly estimated despite smoothing techniqu”
The FLORES Evaluation Datasets for Low-Resource Machine Translation: Nepali–English and Sinhala–English (2019, background): “However, both languages have a rather large amount of monolingual data publicly available (Buck et al., 2014), making them perfect candidates to track performance on unsupervised and semi-supervised tasks for Machine Translation.”
Semantically-enriched 3D models for common-sense knowledge (2015, methodology): “To get these linguistic cues we use recently developed language models [10] that were learned from billions of webpages [3].”
ParaCrawl: Web-Scale Acquisition of Parallel Corpora (2020, methodology): “By splitting this large collection of crawled web pages by web domain and running text extraction and language identiﬁcation (Buck et al., 2014), we can extract statistics on what language content exists on each of them.”
A Massive Collection of Cross-Lingual Web-Document Pairs (2019, methodology): “Previous works have leveraged the data from Common Crawl for mining ngram counts to perform language modeling (Buck et al., 2014).”