Beyond BLEU:Training Neural Machine Translation with Semantic Similarity

By John Wieting, Taylor Berg-Kirkpatrick, Kevin Gimpel, Graham Neubig. 2019

188 citations recorded by the source corpus.

While most neural machine translation (NMT)systems are still trained using maximum likelihood estimation, recent work has demonstrated that optimizing systems to directly improve evaluation metrics such as BLEU can significantly improve final translation accuracy. However, training with BLEU has some limitations: it doesn’t assign partial credit, it has a limited range of output values, and it can penalize semantically correct hypotheses if they differ lexically from the reference. In this paper, we introduce an alternative reward function for optimizing NMT systems that is based on recent work in semantic similarity. We evaluate on four disparate languages trans-lated to English, and find that training with our proposed metric results in better translations as evaluated by BLEU, semantic similarity, and human evaluation, and also that the optimization procedure converges faster. Analysis suggests that this is because the proposed metric is more conducive to optimization, assigning partial credit and providing more diversity in scores than BLEU

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, background): “They even find this to be true when the reward signal is semantic similarity instead of BLEU (Wieting et al., 2019).”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “ text summarization (Gabriel et al., 2019). In future work, we look forward to designing new task-speciﬁc metrics that use BERTSCORE as a subroutine and accommodate task-speciﬁc needs, similar to how Wieting et al. (2019) suggests to use semantic similarity for machine translation training. Because ”
BRIO: Bringing Order to Abstractive Summarization (2022, methodology): “Minimum risk training (Shen et al., 2016; Wieting et al., 2019) and other online sampling based methods (Bengio et al.”
Learning an Unreferenced Metric for Online Dialogue Evaluation (2020, background): “trained on the metric (Wieting et al., 2019), by human evaluation.”
SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization (2021, methodology): “Minimum risk training, as an alternative, has also been used in the language generation tasks (Shen et al., 2016; Wieting et al., 2019).”
A Survey of Evaluation Metrics Used for NLG Systems (2020, unknown): “ed word embeddings. However, recently pretrained transformer models are also being used in a few metrics. SIMILE: To facilitate better comparison of hypothesis and reference sentences, Wieting et al. [168] train a sentence encoder, �, on a set of paraphrase pairs (from ParaNMT corpus [169]) using th”
A Bilingual Generative Transformer for Semantic Sentence Embedding (2019, methodology): “For instance, some of the top performing techniques use word embedding averaging (Wieting et al., 2016b), character n-grams (Wieting et al., 2016a), and subword embedding averaging (Wieting et al., 2019b) to create representations.”
Text Detoxification using Large Pre-trained Neural Models (2021, methodology): “Content preservation (SIM) is evaluated as the similarity of sentence-level embeddings of the original and transformed texts computed by the model of Wieting et al. (2019).”
Modeling Fluency and Faithfulness for Diverse Neural Machine Translation (2019, methodology): “Wieting et al. (2019) used a semantic similarity loss as a reward to measure the similarity between the embedding of the generated translation and the reference.”
Reformulating Unsupervised Style Transfer as Paraphrase Generation (2020, methodology): “We use the following ﬁltering, Content Filtering : We remove all sentence pairs which score lower than 0.5 on a strong paraphrase similarity model from Wieting et al. (2019).”
Confident Adaptive Language Modeling (2022, background): “Training for global consistency, however, could be challenging [81] as it depends on possibly noisy signals that might affect the learning, and also breaks the efﬁcient teacher-forcing training strategy of LMs that relies on local-decisions.”
Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defense (2023, methodology): “Align sentences of p to sentences of q by using the semantic similarity scores from the paraphrase similarity metric in Wieting et al. [2019] to run the sequence alignment algorithm designed by Needleman and Wunsch [1970] which uses dynamic programming (metric details in Section 4.1).”
Reward Optimization for Neural Machine Translation with Learned Metrics (2021, background): “Wieting et al. (2019) and Schmidt and Hofmann (2020) proposed metrics based on the similarity between sentence embeddings, and optimize the text generation models with the similarity scores.”
Revisiting the Weaknesses of Reinforcement Learning for Neural Machine Translation (2021, background): “objective due to the renormalization of model scores, but that has not hindered its empirical success (Shen et al., 2016; Edunov et al., 2018; Wieting et al., 2019; Wang and Sennrich, 2020).”
Controlled Text Generation as Continuous Optimization with Multiple Constraints (2021, methodology): “Following the baseline model Krishna et al. [28], we evaluate the generated sentences with the following metrics: (a) ﬂuency or grammatical wellformedness measured by the accuracy of a RoBERTa-based classiﬁer model [38] trained on CoLA [62], averaged over all outputs, (b) transfer : measured by a Ro”