PMIndia - A Collection of Parallel Corpora of Languages of India

By Barry Haddow, Faheem Kirefu. 2020

122 citations recorded by the source corpus.

Parallel text is required for building high-quality machine translation (MT) systems, as well as for other multilingual NLP applications. For many South Asian languages, such data is in short supply. In this paper, we described a new publicly available corpus (PMIndia) consisting of parallel sentences which pair 13 major languages of India with English. The corpus includes up to 56000 sentences for each language pair. We explain how the corpus was constructed, including an assessment of two different automatic sentence alignment methods, and present some initial NMT results on the corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Machine Translation

Citations in the corpus

MURAL: Multimodal, Multitask Retrieval Across Languages (2021, background): “0, Autshumato (Groe-newald and du Plooy, 2010), PMIndia (Haddow and Kirefu, 2020), CVIT (Srivastava et al., 2020), Inuktitut (Hernandez and Nguyen, 2020), NLPC, JESC (Pryzant et al., 2018), KFTT (Neubig, 2011), ASPEC (Nakazawa et al., 2016), Flores (Guzmán et al., 2019).”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, background): “, 2020), PMIndia V1 (Haddow and Kirefu, 2020), SIPC (Post et al.”
Building Machine Translation Systems for the Next Thousand Languages (2022, background): “Datasets for lower-resourced languages of India have also sprung up, including the 13-language PMIndia (Haddow & Kirefu, 2020), and datasets focused on languages of the Northeast like Mizo (Thihlum et al., 2020), Khasi (Laskar et al., 2021) and Assamese (Laskar et al., 2020).”
Findings of the 2020 Conference on Machine Translation (WMT20) (2020, methodology): “The released parallel HI–MR dataset was collected from news (Siripragada et al., 2020), PMIndia (Haddow and Kirefu, 2020) and Indic Wordnet (Bhattacharyya, 2010; Kunchukuttan, 2020a) datasets.”
Smart Radio Environments Empowered by Reconfigurable Intelligent Surfaces: How It Works, State of Research, and The Road Ahead (2020, background): “In [219], the authors consider a NOMA-based system in which a base station transmits a multiplex of signals to multiple users by means of an RIS.”
Findings of the 2021 Conference on Machine Translation (WMT21) (2021, background): “The released parallel Tamil–Telugu dataset was collected from news (Siripragada et al., 2020), PMIndia (Haddow and Kirefu, 2020) and MKB (Man Ki Baat) datasets.”
A Survey of Multilingual Neural Machine Translation (2019, methodology): “For Indic languages, the CVIT-PIB corpus [136], the PMIndia corpus [60], and the IndoWordNet parallel corpus [84] have also been released recently.”