Building Large Monolingual Dictionaries at the Leipzig Corpora Collection: From 100 to 200 Languages

By Dirk Goldhahn, Thomas Eckart, Uwe Quasthoff. International Conference on Language Resources and Evaluation, 2012

521 citations recorded by the source corpus.

The Leipzig Corpora Collection offers free online access to 136 monolingual dictionaries enriched with statistical information. In this paper we describe current advances of the project in collecting and processing text data automatically for a large number of languages. Our main interest lies in languages of “low density”, where only few text data exists online. The aim of this approach is to create monolingual dictionaries and statistical information for a high number of new languages and to expand the existing dictionaries, opening up new possibilities for linguistic typology and other research. Focus of this paper will be set on the infrastructure for the automatic acquisition of large amounts of monolingual text in many languages from various sources. Preliminary results of the collection of text data will be presented. The mainly language-independent framework for preprocessing, cleaning and creating the corpora and computing the necessary statistics will also be depicted.

Subfields: Language and Linguistics

View Paper: https://www.semanticscholar.org/paper/1b560f892432fb853d233c92f9294640bc91de3c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Language and Linguistics

1. Anna Rogers
2. Yulia Tsvetkov
3. Iryna Gurevych
4. Orhan Firat
5. Mitesh M. Khapra
6. Ricardo Usbeck
7. Ivan Habernal

Citations in the corpus

• iNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020, methodology): “We analyzed whether we could augment our crawls with data from other smaller sources like Leipzig corpus (Goldhahn et al., 2012), WMT NewsCrawl, WMT CommonCrawl (Buck et al., 2014), HindEnCorp (Hindi) (Bojar et al., 2014), etc .”
• C4Corpus: Multilingual Web-size Corpus with Free License (2016, background): “Second, they are usually available as either n-grams (Brants and Franz, 2006) or shuffled sentences, e.g., COW (Sch¨afer and Bildhauer, 2012) or Leipzig Corpora (Goldhahn et al., 2012).”
• AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages (2020, methodology): “pus (Goldhahn et al., 2012) (Tamil and Bengali), WMT NewsCrawl (for Hindi), WMT Common-”
• Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020, background): “The Corpora Collection at Leipzig University (Goldhahn et al., 2012) and the Corpus of Global Language Use (Dunn, 2020) offer corpora in 252 and 148 languages.”
• Building Machine Translation Systems for the Next Thousand Languages (2022, background): “A variety of web-derived collections for hundreds of languages is available for anyone to download, such as the Corpora Collection at Leipzig University (Goldhahn et al., 2012), the Corpus of Global Language Use (Dunn, 2020), ParaCrawl (Esplà et al.”
• AraWEAT: Multidimensional Analysis of Biases in Arabic Word Embeddings (2020, methodology): “…technology for Arabic have the potential for affecting more people than for most other languages; (2) language resources for Arabic – large corpora (Goldhahn et al., 2012), pretrained word embeddings (Mohammad et al., 2017; Bojanowski et al., 2017), and datasets for measuring semantic quality…”
• Resources and benchmark corpora for hate speech detection: a systematic review (2020, methodology): “…DeWaC corpora (Baroni et al. 2009); for English, we compute the word frequencies from the British National Corpus (Clear 1993); for Spanish we compute the word frequencies from the Spanish Billion Word corpus (Cardellino 2016); for Hindi we use the Leipzig corpora collection (Goldhahn et al. 2012).”
• The SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion (2020, methodology): “2017), words were sampled according to their frequency in the largest available Wortschatz (Goldhahn et al. 2012) corpus for that language.”
• N³ - A Collection of Datasets for Named Entity Recognition and Disambiguation in the NLP Interchange Format (2014, methodology): “This corpus has been created using a dataset comprising a list of 1,457 RSS feeds as compiled in (Goldhahn et al., 2012).”
• The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “Leipzig Corpora Collection ur Goldhahn et al. (2012) Mann Ki Baat bn, gu, hi, ml, mr, or, ta, te, ur Siripragada et al.”
• Ranking Transfer Languages with Pragmatically-Motivated Features for Multilingual Sentiment Analysis (2020, methodology): “(Daille, 1994) to extract top-k MWE from a large news-crawl corpus (Goldhahn et al., 2012).”