Writing System and Speaker Metadata for 2,800+ Language Varieties
By Daan van Esch, Tamar Lucassen, Sebastian Ruder, Isaac Caswell, Clara Rivera. 2022
33 citations recorded by the source corpus.
We describe an open-source dataset providing metadata for about 2,800 language varieties used in the world today. Specifically, the dataset provides the attested writing system(s) for each of these 2,800+ varieties, as well as an estimated speaker count for each variety. This dataset was developed through internal research and has been used for analyses around language technologies. This is the largest publicly-available, machine-readable resource with writing system and speaker information for the world’s languages. We analyze the distribution of languages and writing systems in our data and compare it to their representation in current NLP. We hope the availability of this data will catalyze research in under-represented languages.
Subfields: Artificial Intelligence, Linguistics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Orhan Firat
- Ankur Bapna
- Graham Neubig
- Quoc V. Le
- Yu Zhang
- Yonghui Wu
- Mohit Bansal
- Luke S. Zettlemoyer
In Linguistics
- Jacob Eisenstein
- Timothy Dozat
- Alexis Conneau
- Dan Garrette
- Sebastian Gehrmann
- Isaac Caswell
- Daan van Esch
Citations in the corpus
- FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, background): “Table 9: Characteristics of the 102 languages in FLEURS , with their ISO codes, language families, estimated number of speakers in millions (#S) from [39].”
- Building Machine Translation Systems for the Next Thousand Languages (2022, background): “For example, Google Translate supports Frisian, Maltese, Icelandic, and Corsican, each with fewer than 1M L1 speakers, but not (up until this work) Bhojpuri (~51M speakers), Oromo (~24M speakers), Quechua (~9M speakers), or Tigrinya (~9M speakers) (van Esch et al., 2022).”
- Dialect-robust Evaluation of Generated Text (2022, background): “The development of a datagathering approach for subnational and transnational dialects is an important topic for future work. lowing the classification by van Esch et al. (2022).9 Given a sentence, we balance the ratio of sampling a dialect or language tag using a parameter λ.”