Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR

By Juri Opitz, Anette Frank. 2020

39 citations recorded by the source corpus.

Systems that generate natural language text from abstract meaning representations such as AMR are typically evaluated using automatic surface matching metrics that compare the generated texts to reference texts from which the input meaning representations were constructed. We show that besides well-known issues from which such metrics suffer, an additional problem arises when applying these metrics for AMR-to-text evaluation, since an abstract meaning representation allows for numerous surface realizations. In this work we aim to alleviate these issues by proposing \mathcal{M}\mathcal{F}_\beta, a decomposable metric that builds on two pillars. The first is the principle of meaning preservation \mathcal{M} : it measures to what extent a given AMR can be reconstructed from the generated sentence using SOTA AMR parsers and applying (fine-grained) AMR evaluation metrics to measure the distance between the original and the reconstructed AMR. The second pillar builds on a principle of (grammatical) form \mathcal{F} that measures the linguistic quality of the generated text, which we implement using SOTA language models. In two extensive pilot studies we show that fulfillment of both principles offers benefits for AMR-to-text evaluation, including explainability of scores. Since \mathcal{M}\mathcal{F}_\beta does not necessarily rely on gold AMRs, it may extend to other text generation tasks.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

The GEM Benchmark: Natural Language Generation, its Evaluation and Metrics (2021, background): “grained and interpretable evaluation metrics, for example to measure consistency in data-to-text problems (Opitz and Frank, 2020; Dhingra et al., 2019).”
Data-QuestEval: A Referenceless Metric for Data-to-Text Semantic Evaluation (2021, methodology): “Beyond n-grams, Opitz and Frank (2020) proposed to use a model trained on the reverse task, i.e. text to data reconstruction, and to compare the two data generated i) from the reference, and ii) from the hypothesis.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “Some very recent works in the direction of interpretability of metrics are proposed in [115, 120].”