Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus

By Isaac Caswell, Theresa Breiner, Daan van Esch, Ankur Bapna. 2020

112 citations recorded by the source corpus.

Large text corpora are increasingly important for a wide variety of Natural Language Processing (NLP) tasks, and automatic language identification (LangID) is a core technology needed to collect such datasets in a multilingual context. LangID is largely treated as solved in the literature, with models reported that achieve over 90% average F1 on as many as 1,366 languages. We train LangID models on up to 1,629 languages with comparable quality on held-out test sets, but find that human-judged LangID accuracy for web-crawl text corpora created using these models is only around 5% for many lower-resource languages, suggesting a need for more robust evaluation. Further analysis revealed a variety of error modes, arising from domain mismatch, class imbalance, language similarity, and insufficiently expressive models. We propose two classes of techniques to mitigate these errors: wordlist-based tunable-precision filters (for which we release curated lists in about 500 languages) and transformer-based semi-supervised LangID models, which increase median dataset precision from 5.5% to 71.2%. These techniques enable us to create an initial data set covering 100K or more relatively clean sentences in each of 500+ languages, paving the way towards a 1,000-language web text corpus.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, background): “Text on modern digital platforms is noisy and exhibits complex character-level phenomena such as typos, character repetitions, and non-standard case changes (Caswell et al., 2020).”
Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling (2023, background): “Existing massive multilingual datasets can be of dubious quality (Caswell et al., 2020; Kreutzer et al., 2021) and we do not feel qualified to vet existing multilingual datasets well enough to determine issues that may arise due to using them.”
Quality at a Glance: An Audit of Web-Crawled Multilingual Datasets (2021, methodology): “In highly multilingual settings, past work found that web-crawls of lowerresource languages have serious issues, especially with segment-level LangID (Caswell et al., 2020).”
Trafilatura: A Web Scraping Library and Command-Line Tool for Text Discovery and Extraction (2021, methodology): “As useful monolingual text corpora across languages are highly relevant for the NLP community (Caswell et al., 2020), web corpora seem to be a natural way to gather language data.”
Building Machine Translation Systems for the Next Thousand Languages (2022, methodology): “, 2016) and Semi-Supervised LangID (SSLID) models (Caswell et al., 2020).”
Ungoliant: An Optimized Pipeline for the Generation of a Very Large-Scale Multilingual Web Corpus (2021, unknown): “While OSCAR is considered to be one of the cleanest multilingual corpora available (Caswell et al., 2020, 2021), several problems have been described, and the state of the publicly available code raises questions about maintenance and maintenability of the pipeline itself.”
TaTa: A Multilingual Table-to-Text Dataset for African Languages (2022, methodology): “The model was trained using the same hyper-parameters as mT5, using mC4 (Xue et al., 2021) and additional automatically mined data in around 250 SSA languages (Caswell et al., 2020).”
Towards a Cleaner Document-Oriented Multilingual Crawled Corpus (2022, background): “, 2020) there a many concerns that still need to be tackled (Caswell et al., 2020) specially for multilingual data (Caswell et al.”
OBELISC: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documents (2023, background): “Data audits have shed light on the some limitations and unintended biases contained in these text corpora (Caswell et al., 2020; Dodge et al., 2021).”