Mining the Correlation between Human and Automatic Evaluation at Sentence Level
By Yanli Sun. 2010
12 citations recorded by the source corpus.
Automatic evaluation metrics are fast and cost-effective measurements of the quality of a Machine Translation (MT) system. However, as humans are the end-user of MT output, human judgement is the benchmark to assess the usefulness of automatic evaluation metrics. While most studies report the correlation between human evaluation and automatic evaluation at corpus level, our study examines their correlation at sentence level. In addition to the statistical correlation scores, such as Spearman's rank-order correlation coefficient, a finer-grained and detailed examination of the sensitivity of automatic metrics compared to human evaluation is also reported in this study. The results show that the threshold for human evaluators to agree with the judgements of automatic metrics varies with the automatic metrics at sentence level. While the automatic scores for two translations are greatly different, human evaluators may consider the translations to be qualitatively similar and vice versa. The detailed analysis of the correlation between automatic and human evaluation allows us determine with increased confidence whether an increase in the automatic scores will be agreed by human evaluators or not.
Subfields: Artificial Intelligence, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Evaluation & Benchmarks
Citations in the corpus
- Survey of Hallucination in Natural Language Generation (2022, background): “There are two main forms of human evaluation: (1) scoring, where human annotators rate the hallucination level in a range; and (2) comparing, where human annotators compare the output texts with baselines or ground-truth references [176].”