Correlating Human and Automatic Evaluation of a German Surface Realiser

By Aoife Cahill. 2009

30 citations recorded by the source corpus.

We examine correlations between native speaker judgements on automatically generated German text against automatic evaluation metrics. We look at a number of metrics from the MT and Summarisation communities and find that for a relative ranking task, most automatic metrics perform equally well and have fairly strong correlations to the human judgements. In contrast, on a naturalness judgement task, the General Text Matcher (GTM) tool correlates best overall, although in general, correlation between the human judgements and the automatic metrics was quite weak.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Towards Coherent and Engaging Spoken Dialog Response Generation Using Automatic Conversation Evaluators (2019, background): “While the shortcomings of these metrics are well known for MT (Graham, 2015; Espinosa et al., 2010; Cahill, 2009), the problem is aggravated for dialog generation evaluation because of the much larger output space (Liu et al., 2016).”
How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generation (2016, background): “, 2015) and natural language generation (NLG) (Stent et al., 2005; Cahill, 2009; Reiter and Belz, 2009; Espinosa et al., 2010).”
Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation (2017, background): “By contrast, Espinosa et al. (2010) found that automatic metrics such as nist, meteor and gtm correlate moderately well with human fluency and adequacy judgements of English surface realisation quality, while Cahill (2009) reported only a weak correlation for German surface realisation.”
On Evaluating and Comparing Conversational Agents (2018, background): “There is work in MT and Natural Language Generation (NLG) ﬁelds which studies correlation of BLEU with human judgement and highlights some of its shortcomings (Graham, 2015; Espinosa et al., 2010; Cahill, 2009).”
On Evaluating and Comparing Open Domain Dialog Systems (2018, background): “There is work in MT and Natural Language Generation (NLG) fields which studies correlation of BLEU with human judgement and highlights some of its shortcomings (Graham, 2015; Espinosa et al., 2010; Cahill, 2009).”
Mining the Correlation between Human and Automatic Evaluation at Sentence Level (2010, background): “Cahill, A. (2009). 'Correlating Human and Automatic Evaluation of a German Surface Realiser'. In Proceedings of the ACL-IJNLP 2009 Conference Short Papers, Suntec, Singapore, August, pp. 97-100. Callison-Burch, C., Fordyce, C., Koehn, P., Monz, C., & Schroeder, J. (2008). 'Further Meta-evaluation of”