HindEnCorp - Hindi-English and Hindi-only Corpus for Machine Translation

By Ondrej Bojar, Vojtech Diatka, Pavel Rychlý, P. Straňák, Vít Suchomel, Ales Tamchyna, Daniel Zeman. 2014

133 citations recorded by the source corpus.

We present HindEnCorp, a parallel corpus of Hindi and English, and HindMonoCorp, a monolingual corpus of Hindi in their release version 0.5. Both corpora were collected from web sources and preprocessed primarily for the training of statistical machine translation systems. HindEnCorp consists of 274k parallel sentences (3.9 million Hindi and 3.8 million English tokens). HindMonoCorp amounts to 787 million tokens in 44 million sentences. Both the corpora are freely available for non-commercial research and their preliminary release has been used by numerous participants of the WMT 2014 shared translation task.

Subfields: Artificial Intelligence, Language and Linguistics, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Language and Linguistics

In Natural Language Processing

Citations in the corpus

Edinburgh’s Phrase-based Machine Translation Systems for WMT-14 (2014, methodology): “Interpolation: We trained two phrase translation tables p(ūi|ēi) and p(ēi|h̄i), from UrduEnglish (Indic corpus) and Hindi-English (HindEnCorp (Bojar et al., 2014)) bilingual corpora.”
Findings of the 2014 Workshop on Statistical Machine Translation (2014, background): “…CzEng, Common Crawl, Russian-English Wikipedia Headlines provided by CMU), some were updated (Russian-English parallel data provided by Yandex, News Commentary, monolingual data), and a new corpus was added (Hindi-English corpus, Bojar et al. (2014)), Hindi-English Wikipedia Headline corpus).”
DCU-Lingo24 Participation in WMT 2014 Hindi-English Translation task (2014, methodology): “We use HindEnCorp 8 (Bojar et al., 2014) parallel corpus for MT system building.”
CUNI in WMT14: Chimera Still Awaits Bellerophon (2014, methodology): “We used HindEnCorp (Bojar et al., 2014) as the sole source of parallel data (nearly 276 thousand sentence pairs, around 3.95 million English tokens and 4.09 million Hindi tokens).”
AI4Bharat-IndicNLP Corpus: Monolingual Corpora and Word Embeddings for Indic Languages (2020, methodology): “, 2014) (Tamil, Malayalam), HindEnCorp (Hindi) (Bojar et al., 2014).”
Machine Translation and Monolingual Postediting: The AFRL WMT-14 System (2014, methodology): “The HindEnCorp corpus (Bojar et al., 2014) is distributed in tokenized form; in order to ensure a uniform tokenization standard across all of our data, we began by detokenized this data using the Moses detokenization scripts.”
Manawi: Using Multi-Word Expressions and Named Entities to Improve Machine Translation (2014, methodology): “To train the monolingual language model, we used the Hindi sentences from the HindEnCorp.”
Improving machine translation via triangulation and transliteration (2014, unknown): “The Hindi-English systems were trained using Hindi-English parallel data (Bojar et al., 2014) composed by compiling several sources including the Hindi-English segment of the Indic parallel corpus.”
Generating High Quality Proposition Banks for Multilingual Semantic Role Labeling (2015, unknown): “, 2009), Europarl: (Koehn, 2005), Hindencorp: (Bojar et al., 2014).”
iNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (2020, methodology): “We analyzed whether we could augment our crawls with data from other smaller sources like Leipzig corpus (Goldhahn et al., 2012), WMT NewsCrawl, WMT CommonCrawl (Buck et al., 2014), HindEnCorp (Hindi) (Bojar et al., 2014), etc .”