Perturbation CheckLists for Evaluating NLG Evaluation Metrics

By Ananya B. Sai, Tanay Dixit, Dev Yashpal Sheth, Sreyas Mohan, Mitesh M. Khapra. 2021

69 citations recorded by the source corpus.

Natural Language Generation (NLG) evaluation is a multifaceted task requiring assessment of multiple desirable criteria, e.g., fluency, coherency, coverage, relevance, adequacy, overall quality, etc. Across existing datasets for 6 NLG tasks, we observe that the human evaluation scores on these multiple criteria are often not correlated. For example, there is a very low correlation between human scores on fluency and data coverage for the task of structured data to text generation. This suggests that the current recipe of proposing new automatic evaluation metrics for NLG by showing that they correlate well with scores assigned by humans for a single criteria (overall quality) alone is inadequate. Indeed, our extensive study involving 25 automatic evaluation metrics across 6 different tasks and 18 different evaluation criteria shows that there is no single metric which correlates well with human scores on all desirable criteria, for most NLG tasks. Given this situation, we propose CheckLists for better design and evaluation of automatic metrics. We design templates which target a specific criteria (e.g., coverage) and perturb the output such that the quality gets affected only along this specific criteria (e.g., the coverage drops). We show that existing evaluation metrics are not robust against even such simple perturbations and disagree with scores assigned by humans to the perturbed output. The proposed templates thus allow for a fine-grained assessment of automatic evaluation metrics exposing their limitations and will facilitate better design, analysis and evaluation of such metrics. Our templates and code are available at https://iitmnlp.github.io/EvalEval/

Subfields: Artificial Intelligence, Information Systems, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “The current empirical evidence for this is negative —in section 3.4 we will present several studies that show that even current metrics break down with simple adversarial examples (Sai et al., 2021; Kaster et al., 2021).”
FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation (2022, background): “For instance, Sai et al. (2021) analyzes the human ratings with respect to (w.r.t.) different ﬁne-grained dimensions on four text generation tasks and has observed moderate correlations for most dimension pairs.”
Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implications (2022, background): “This difficulty is often compounded by the need for NLG systems to meet a variety of goals— often measured in competing and imperfect ways— depending on their deployment settings (Sai et al., 2021).”
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022, unknown): “Similar to the previous findings (Sai et al., 2021), BERTScore suffers greatly when word order is shuffled and fails to capture the shifts in semantic meanings.”