The statistical advantage of automatic NLG metrics at the system level

By Johnny Tian-Zheng Wei, Robin Jia. 2021

25 citations recorded by the source corpus.

Estimating the expected output quality of generation systems is central to NLG. This paper qualifies the notion that automatic metrics are not as good as humans in estimating system-level quality. Statistically, humans are unbiased, high variance estimators, while metrics are biased, low variance estimators. We compare these estimators by their error in pairwise prediction (which generation system is better?) using the bootstrap. Measuring this error is complicated: predictions are evaluated against noisy, human predicted labels instead of the ground truth, and metric predictions fluctuate based on the test sets they were calculated on. By applying a bias-variance-noise decomposition, we adjust this error to a noise-free, infinite test set setting. Our analysis compares the adjusted error of metrics to humans and a derived, perfect segment-level annotator, both of which are unbiased estimators dependent on the number of judgments collected. In MT, we identify two settings where metrics outperform humans due to a statistical advantage in variance: when the number of human judgments used is small, and when the quality difference between compared systems is small.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “Wei and Jia (2021) show that the requirement for more judgments can quickly becomes prohibitive: to detect a difference of 1 point on a 1-100 scale in WMT, we need 10,000 perfect annotator judgments.”
Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluation (2022, background): “, 2021), with some work noting that thousands of costly data points may need to be collected in order to draw statistically significant conclusions (Wei and Jia, 2021).”