Processing South Asian Languages Written in the Latin Script: the Dakshina Dataset
By Brian Roark, Lawrence Wolf-Sonkin, Christo Kirov, Sabrina J. Mielke, Cibu Johny, Isin Demirsahin, Keith B. Hall. 2020
94 citations recorded by the source corpus.
This paper describes the Dakshina dataset, a new resource consisting of text in both the Latin and native scripts for 12 South Asian languages. The dataset includes, for each language: 1) native script Wikipedia text; 2) a romanization lexicon; and 3) full sentence parallel data in both a native script of the language and the basic Latin alphabet. We document the methods used for preparation and selection of the Wikipedia text in each language; collection of attested romanizations for sampled lexicons; and manual romanization of held-out sentences from the native script collections. We additionally provide baseline results on several tasks made possible by the dataset, including single word transliteration, full sentence transliteration, and language modeling of native script and romanized text.
Subfields: Artificial Intelligence, Language and Linguistics
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ankur Bapna
- Sebastian Ruder
- Isaac Caswell
- Daan van Esch
- Partha P. Talukdar
- Clara Rivera
- Simran Khanuja
- Savya Khosla
In Language and Linguistics
Citations in the corpus
- Language ID in the Wild: Unexpected Challenges on the Path to a Thousand-Language Web Text Corpus (2020, background): “And many languages that are not officially written in the Latin alphabet have informal transliterated orthographies (Roark et al., 2020); our models can identify the most common ones, but we could cover more.”
- MuRIL: Multilingual Representations for Indian Languages (2021, methodology): “First, we use the Dakshina Dataset (Roark et al., 2020) that contains 10,000 sentence pairs for 12 IN languages (bn, gu, hi, kn, ml, mr, pa, ta, te, ur).”
- Writing System and Speaker Metadata for 2,800+ Language Varieties (2022, background): “…have been shown to perform poorly on languages with limited amount of pre-training data and on languages with non-Latin scripts (Hu et al., 2020; Pfeiffer et al., 2021; Muller et al., (Bhat et al., 2014) or transliterate 8 libraries, or the manually curated Dakshina dataset (Roark et al., 2020).”