The IIT Bombay English-Hindi Parallel Corpus

By Anoop Kunchukuttan, Pratik Mehta, Pushpak Bhattacharyya. 2017

284 citations recorded by the source corpus.

We present the IIT Bombay English-Hindi Parallel Corpus. The corpus is a compilation of parallel corpora previously available in the public domain as well as new parallel corpora we collected. The corpus contains 1.49 million parallel segments, of which 694k segments were not previously available in the public domain. The corpus has been pre-processed for machine translation, and we report baseline phrase-based SMT and NMT translation results on this corpus. This corpus has been used in two editions of shared tasks at the Workshop on Asian Language Translation (2016 and 2017). The corpus is freely available for non-commercial research. To the best of our knowledge, this is the largest publicly available English-Hindi parallel corpus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Multilingual Denoising Pre-training for Neural Machine Translation (2020, methodology): “We also use FLoRes pairs (Guzm´an et al., 2019, En-Ne and En-Si), En-Hi from IITB (Kunchukuttan et al., 2017), and En-My from WAT19 (Ding et al., 2018, 2019).”
Cross-lingual Language Model Pretraining (2019, methodology): “Precisely, we use MultiUN [44] for French, Spanish, Russian, Arabic and Chinese, and the IIT Bombay corpus [3] for Hindi.”
XNLI: Evaluating Cross-lingual Sentence Representations (2018, methodology): “…Russian, Arabic and Chinese, we use the United Nation corpora (Ziemski et al., 2016), for German, Greek and Bulgarian, the Europarl corpora (Koehn, 2005), for Turkish, Vietnamese and Thai, the OpenSubtitles 2018 corpus (Tiedemann, 2012), and for Hindi, the IIT Bombay corpus (Anoop et al., 2018).”
The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Dataset (2023, background): “(2021) Indosum id Kurniawan and Louvan (2018) KALIMAT ar El-Haj and Koulali (2013) KSUCCA ar Alrabiah et al.”
Unicoder: A Universal Language Encoder by Pre-training with Multiple Cross-lingual Tasks (2019, methodology): “We use same MT dataset as Lample and Conneau (2019) which are collected from MultiUN (Eisele and Chen, 2010), IIT Bombay corpus (Kunchukuttan et al., 2017), OpenSubtitles 2018, EUbookshop corpus and GlobalVoices.”
InfoXLM: An Information-Theoretic Framework for Cross-Lingual Language Model Pre-Training (2020, methodology): “We collect translation pairs from MultiUN (Ziemski et al., 2016), IIT Bom-bay (Kunchukuttan et al., 2018), OPUS (Tiede-mann, 2012), and WikiMatrix (Schwenk et al., 2019).”
Samanantar: The Largest Publicly Available Parallel Corpora Collection for 11 Indic Languages (2021, unknown): “We also collated parallel data from the following non-OPUS sources (URLs can be found in the Indic NLP Catalog): ALT (Riza et al., 2016), BanglaNMT (Hasan et al., 2020), CVITPIB (Philip et al., 2020), IITB (Kunchukuttan et al., 2018), MTEnglish2Odia, NLPC, OdiEnCorp 2.0 (Parida et al., 2020), PMIndi”
Socially Aware Bias Measurements for Hindi Language Representations (2021, unknown): “…2014; Bojanowski et al., 2017; Devlin et al., 2019) are commonly used across the world in a variety of applications including machine translation (Kunchukuttan et al., 2017), information retrieval (Rao and Devi, 2018), chatbots (Bhag-wat et al., 2019), sentiment classiﬁcation (Kumar et al.,…”
Hindi-English Hate Speech Detection: Author Profiling, Debiasing, and Practical Perspectives (2020, methodology): “The preprocessing was a ﬁve-step process before the preprocessing steps deﬁned in the previous subsection: • Two separate sets of Hindi and English words were formed using the NLTK English dictionary and the IIT Bombay Hindi dictionary (Kunchukuttan, Mehta, and Bhattacharyya 2017).”
Cross-Lingual Training for Automatic Question Generation (2019, background): “1 Back translation has been demonstrated to be very important for un-supervised NMT (Yang et al., 2018; Lample et al., 2018).”
ERNIE-M: Enhanced Multilingual Representation by Aligning Cross-lingual Semantics with Monolingual Corpora (2020, methodology): “We use the Adam optimizer (Kingma and Ba, 2014) to train E RNIE -”
PMIndia - A Collection of Parallel Corpora of Languages of India (2020, background): “The largest parallel corpus we are aware of for South Asian languages is the IIT Bombay English–Hindi corpus (Kunchukuttan et al., 2018), containing about 1.5M sentence pairs from various sources.”
Alternating Language Modeling for Cross-Lingual Pre-Training (2020, methodology): “Hindi data is from the IIT Bombay corpus (Kunchukuttan, Mehta, and Bhattacharyya 2018).”
mT6: Multilingual Pretrained Text-to-Text Transformer with Translation Pairs (2021, methodology): “, 2016), IIT Bombay (Kunchukuttan et al., 2018), OPUS (Tiedemann, 2012), and WikiMatrix (Schwenk et al.”
Evaluating Gender Bias in Hindi-English Machine Translation (2021, methodology): “(2021) was trained on the IIT-Bombay Hindi-English parallel data corpus (Kunchukuttan et al., 2018), which contains approximately 1.”
On Learning Universal Representations Across Languages (2020, methodology): “For bilingual data, we use the same (English-to-X) MT dataset as (Conneau & Lample, 2019), which are collected from MultiUN (Eisele & Yu, 2010) for French, Spanish, Arabic and Chinese, the IIT Bombay corpus (Kunchukuttan et al., 2018) for Hindi, the OpenSubtitles 2018 for Turkish, Vietnamese and Tha”