Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis

By Wenda Xu, Yi-Lin Tuan, Yujie Lu, Michael Stephen Saxon, Lei Li, William Yang Wang. 2022

25 citations recorded by the source corpus.

Is it possible to build a general and automatic natural language generation (NLG) evaluation metric? Existing learned metrics either perform unsatisfactorily or are restricted to tasks where large human rating data is already available. We introduce SESCORE, a model-based metric that is highly correlated with human judgements without requiring human annotation, by utilizing a novel, iterative error synthesis and severity scoring pipeline. This pipeline applies a series of plausible errors to raw text and assigns severity labels by simulating human judgements with entailment. We evaluate SESCORE against existing metrics by comparing how their scores correlate with human ratings. SESCORE outperforms all prior unsupervised metrics on multiple diverse NLG tasks including machine translation, image captioning, and WebNLG text generation. For WMT 20/21 En-De and Zh-En, SESCORE improve the average Kendall correlation with human judgement from 0.154 to 0.195. SESCORE even achieves comparable performance to the best supervised metric COMET, despite receiving no human-annotated training data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Evaluation & Benchmarks, Natural Language Generation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tom Kocmi
2. Ricardo Rei
3. Markus Freitag
4. Alon Lavie
5. Nitika Mathur
6. George F. Foster
7. Craig Alan Stewart
8. Chi-kiu (羅致翹) Lo

In Computer Vision and Pattern Recognition

1. Mingxuan Wang
2. Tom Kocmi
3. George F. Foster
4. Markus Freitag
5. Lei Li
6. Wenda Xu
7. Ricardo Rei
8. William Yang Wang

In Evaluation & Benchmarks

1. Tom Kocmi
2. Markus Freitag
3. Alon Lavie
4. Ricardo Rei
5. George F. Foster
6. Craig Alan Stewart
7. Nitika Mathur
8. Eleftherios Avramidis

In Natural Language Generation

1. Lei Li
2. Wenda Xu
3. William Yang Wang

Citations in the corpus

• SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, methodology): “From the surface form difference of x0 and xj , along with the random word drops, we derive a proposal, P for stratified error synthesis (Xu et al., 2022),”
• Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, background): “Based on the alignment matrix, BERTS CORE returns a precision, recall and F1 score.”