Automatic Text Evaluation through the Lens of Wasserstein Barycenters

By Pierre Colombo, Guillaume Staerman, Chloé Clavel, Pablo Piantanida. 2021

44 citations recorded by the source corpus.

A new metric BaryScore to evaluate text generation based on deep contextualized embeddings (e.g., BERT, Roberta, ELMo) is introduced. This metric is motivated by a new framework relying on optimal transport tools, i.e., Wasserstein distance and barycenter. By modelling the layer output of deep contextualized embeddings as a probability distribution rather than by a vector embedding; this framework provides a natural way to aggregate the different outputs through the Wasserstein space topology. In addition, it provides theoretical grounds to our metric and offers an alternative to available solutions (e.g., MoverScore and BertScore). Numerical evaluation is performed on four different tasks: machine translation, summarization, data2text generation and image captioning. Our results show that BaryScore outperforms other BERT based metrics and exhibits more consistent behaviour in particular for text summarization.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “(2021) and Colombo et al. (2022) expand on these insights and a framework by Djolonga et al.”
InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation (2021, methodology): “Future work includes extending our metrics to new tasks such as SLU (Chapuis et al. 2020, 2021; Dinkar et al. 2020; Colombo, Clavel, and Piantanida 2021), controlled sentence generation (Colombo et al. 2019, 2021b) and multi-modal learning (Colombo et al. 2021a; Garcia et al. 2019).”