Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing

By Brian Thompson, Matt Post. 2020

210 citations recorded by the source corpus.

We propose the use of a sequence-to-sequence paraphraser for automatic machine translation evaluation. The paraphraser takes a human reference as input and then force-decodes and scores an MT system output. We propose training the aforementioned paraphraser as a multilingual NMT system, treating paraphrasing as a zero-shot "language pair" (e.g., Russian to Russian). We denote our paraphraser "unbiased" because the mode of our model's output probability is centered around a copy of the input sequence, which in our case represent the best case scenario where the MT system output matches a human reference. Our method is simple and intuitive, and our single model (trained in 39 languages) outperforms or statistically ties with all prior metrics on the WMT19 segment-level shared metrics task in all languages, excluding Gujarati where the model had no training data. We also explore using our model conditioned on the source instead of the reference, and find that it outperforms every quality estimation as a metric system from the WMT19 shared task on quality estimation by a statistically significant margin in every language pair.

Subfields: Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “An alternative approach by Thompson and Post (2020) uses the probabilities of each modelgenerated token under a paraphrasing model that uses the human reference as input.”
Learning Compact Metrics for MT (2021, neutral): “, 2020a), PRISM (Thompson and Post, 2020), and YISI1 (Thompson and Post, 2020), using the WMT Metrics report.”
GPTScore: Evaluate as You Desire (2023, methodology): “PRISM (Thompson and Post, 2020) is a reference-based evaluation method designed for machine translation with pre-trained paraphrase systems.”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “, unsupervised): BLEU [42] ROUGE-1 and ROUGE-2, ROUGE-L, CHRF [44], PRISM [55], MoverScore [64], BERTScore [63].”
BLEU Might Be Guilty but References Are Not Innocent (2020, methodology): “, 2006) or full sentence paraphrases (Kauchak and Barzilay, 2006; Bawden et al., 2020; Thompson and Post, 2020) have been used to consider multiple references.”
Results of the WMT20 Metrics Shared Task (2020, methodology): “This is also true for PRISM and all the reference-free metrics except COMET-QE.”
SMART: Sentences as Basic Units for Text Evaluation (2022, methodology): “Source-dependent model-based metrics include: (13) PRISM (Thompson and Post, 2020): leverages a zero-shot paraphrasing model and uses probabilities from force-decoding the candidate sum-mary given the source as input; (14) Q 2 (Honovich et al., 2021): employs question generation and question answeri”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “Prism P RISM (Thompson and Post, 2020) is an automatic MT metric which uses a sequence-to-sequence paraphraser to score MT system outputs conditioned on their respective human references.”
Dialect-robust Evaluation of Generated Text (2022, methodology): “We use Prism13 (m39v1 checkpoint) for quality estimation with and without references.”
Paraphrase Generation: A Survey of the State of the Art (2021, background): “Paraphrasing can be play an important role in language understanding tasks, such as question answering (Dong et al., 2017; Zhu et al., 2017), machine translation (Seraj et al., 2015; Thompson and Post, 2020a), and semantic parsing (Berant and Liang, 2014; Cao et al., 2020).”
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (2023, methodology): “…also closely related to segment -level and system -level correlations to human judgments respectively, which have been extensively used in developing evaluation metrics in machine translation (Ma et al., 2019; Thompson and Post, 2020) and summarization (Bhandari et al., 2020; Deutsch et al., 2021).”
Generative Pretraining for Paraphrase Evaluation (2021, methodology): “PRISM (Thompson and Post, 2020) similar to ParaBLEU formulates evaluation as a paraphrasing task.”
Learning to Evaluate Translation Beyond English: BLEURT Submissions to the WMT Metrics 2020 Shared Task (2020, methodology): “The scores for BERT SCORE and PRISM come from Thompson and Post (2020). free and calculates its score by comparing translations only to the source sentence.”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, background): “For all text generation tasks, we include 1) three n-gram, distance-based baseline metrics: BLEU (Papineni et al., 2002), ChrF (Popovi´c, 2015) and TER (Snover et al., 2006); 2) four best performed learned metrics without human ratings: PRISM (Thompson and Post, 2020), BARTScore (Yuan et al., 2021),”
In-context Examples Selection for Machine Translation (2022, methodology): “To quantify the meaning equivalence of the 1-best task-level prompt against random 1-shot examples, we report the percentage of aligned words between the source and reference translation (“% Aligned words”) us-ing fastAlign (Dyer et al., 2013) and the probability of generating the reference translat”
Exploring the Use of Large Language Models for Reference-Free Text Quality Evaluation: A Preliminary Empirical Study (2023, methodology): “Following the settings of previous works, we select baseline metrics from the following widely used metrics accordingly: ROUGE-1 , ROUGE-2 and ROUGE-L (Lin, 2004); BERTScore (Zhang* et al., 2020); MoverScore (Zhao et al., 2019); PRISM (Thompson and Post, 2020); BARTScore and its enhanced versions, B”
Automatic Text Evaluation through the Lens of Wasserstein Barycenters (2021, background): “Due to its appealing geometric properties, it has found many applications in machine learning such as generative models (Arjovsky et al., 2017; Tolstikhin et al., 2018; Gulrajani et al., 2017), domain adaptation (Courty et al., 2017), clustering (Ho et al., 2017; Ye et al., 2017), adversarial…”