A Study of Automatic Metrics for the Evaluation of Natural Language Explanations

By Miruna Clinciu, Arash Eshghi, Helen Wright Hastie. 2021

62 citations recorded by the source corpus.

As transparency becomes key for robotics and AI, it will be necessary to evaluate the methods through which transparency is provided, including automatically generated natural language (NL) explanations. Here, we explore parallels between the generation of such explanations and the much-studied field of evaluation of Natural Language Generation (NLG). Specifically, we investigate which of the NLG evaluation measures map well to explanations. We present the ExBAN corpus: a crowd-sourced corpus of NL explanations for Bayesian Networks. We run correlations comparing human subjective ratings with NLG automatic measures. We find that embedding-based automatic NLG evaluation methods, such as BERTScore and BLEURT, have a higher correlation with human ratings, compared to word-overlap metrics, such as BLEU and ROUGE. This work has implications for Explainable AI and transparent robotic and autonomous systems.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, methodology): “Given that existing automatic metrics often do not correlate well with human judgements of explanation quality (Clinciu et al., 2021; Kayser et al., 2021), we conduct human studies for evaluation.”
Measuring Association Between Labels and Free-Text Rationales (2020, background): “Evaluation We do not report BLEU scores (Pap-ineni et al., 2002), because BLEU and related metrics do not measure plausibility (Camburu et al., 2018; Kayser et al., 2021; Clinciu et al., 2021) or faithfulness (Jacovi and Goldberg, 2020).”
Chain of Explanation: New Prompting Method to Generate Quality Natural Language Explanation for Implicit Hate Speech (2022, methodology): “We then remove the instances that are too hard to reach a consensus based on the same rule in [3].”
Evaluating the Evaluation Metrics for Style Transfer: A Case Study in Multilingual Formality Transfer (2021, background): “Automatic (and human) evaluation processes are well-known problems for the field of Natural Language Generation (Howcroft et al., 2020; Clinciu et al., 2021) and the burgeoning subfield of ST is not immune.”
Are Hard Examples also Harder to Explain? A Study with Human and Model-Generated Explanations (2022, methodology): “Due to the limitations of automatic metrics for evaluating explanation quality (Clinciu et al., 2021), we follow Wiegreffe et al. (2022) to conduct human evaluation of both crowd-sourced and GPT-3 explanations on MTurk based on three attributes – grammaticality , supportive-ness , and…”