WMDO: Fluency-based Word Mover’s Distance for Machine Translation Evaluation

By Julian Chow, Lucia Specia, Pranava Swaroop Madhyastha. 2019

27 citations recorded by the source corpus.

We propose WMDO, a metric based on distance between distributions in the semantic vector space. Matching in the semantic space has been investigated for translation evaluation, but the constraints of a translation’s word order have not been fully explored. Building on the Word Mover’s Distance metric and various word embeddings, we introduce a fragmentation penalty to account for fluency of a translation. This word order extension is shown to perform better than standard WMD, with promising results against other types of metrics.

Subfields: Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Evaluation & Benchmarks

In Machine Translation

Citations in the corpus

DART: Open-Domain Structured Data Record to Text Generation (2020, background): “, 2015), WMDO (Chow et al., 2019), MoverScore (Zhao et al.”
BERTScore: Evaluating Text Generation with BERT (2019, background): “, 2015), WMDO (Chow et al., 2019), and SMS (Clark et al.”
Evaluation of Text Generation: A Survey (2020, background): “Chow et al. (2019) investigate the position of the word tokens in evaluating the fluency of the generated text.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “[22] propose a modified version termedWMDO which additionally introduces a”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, methodology): “, 2019) WMDo (Chow et al., 2019) BERT Embeddings MoverScore (Zhao et al.”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “, 2016) and embedding based metrics (Chow et al., 2019; Kusner et al., 2015; Lo and Wu, 2011; Lo, 2019).”