A Human Judgement Corpus and a Metric for Arabic MT Evaluation

By Houda Bouamor, Hanan Alshikhabobakr, Behrang Mohit, Kemal Oflazer. 2014

26 citations recorded by the source corpus.

We present a human judgments datasetand an adapted metric for evaluation ofArabic machine translation. Our mediumscaledataset is the first of its kind for Arabicwith high annotation quality. We usethe dataset to adapt the BLEU score forArabic. Our score (AL-BLEU) providespartial credits for stem and morphologicalmatchings of hypothesis and referencewords. We evaluate BLEU, METEOR andAL-BLEU on our human judgments corpusand show that AL-BLEU has the highestcorrelation with human judgments. Weare releasing the dataset and software tothe research community.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Machine Translation

Citations in the corpus

VaTeX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Research (2019, methodology): “Thus, following previous methods [8, 68] on collecting parallel pairs, we choose the post-editing annotation strategy.”
A Structured Review of the Validity of BLEU (2018, methodology): “For example, Bouamor et al. (2014) presents correlations for both standard BLEU and a modified version called AL-BLEU.”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, methodology): “7This has incentivized the creation of BLEU variants (Bouamor et al., 2014) that rely on morphological analyzers.”
Synthetic Data Augmentation for Zero-Shot Cross-Lingual Question Answering (2020, background): “In Table 2 we report the BLEU4 scores for QG synth+trans grouped by the language of the question.”
Building an Arabic Machine Translation Post-Edited Corpus: Guidelines and Annotation (2016, methodology): “For Arabic, we cite the effort of Bouamor et al. (2014) who created a medium scale human judgment corpus of Arabic machine translation using the output of six MT systems and a total of 1892 sentences and 22K rankings.”