RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation

By Hiroki Shimanaka, Tomoyuki Kajiwara, Mamoru Komachi. 2018

106 citations recorded by the source corpus.

We introduce the RUSE metric for the WMT18 metrics shared task. Sentence embeddings can capture global information that cannot be captured by local features based on character or word N-grams. Although training sentence embeddings using small-scale translation datasets with manual evaluation is difficult, sentence embeddings trained from large-scale data in other tasks can improve the automatic evaluation of machine translation. We use a multi-layer perceptron regressor based on three types of sentence embeddings. The experimental results of the WMT16 and WMT17 datasets show that the RUSE metric achieves a state-of-the-art performance in both segment- and system-level metrics tasks with embedding features only.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Learning to Compare for Better Training and Evaluation of Open Domain Natural Language Generation Models (2020, methodology): “More recently, embedding similarity based metrics such as HUSE (Shimanaka et al. 2018) and BERTScore (Zhang et al.”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “We also compare our correlations with state-of-the-art metrics, including METEOR++ (Guo et al., 2018), CHR F++ (Popovi´c, 2017), BEER (Stano-jevi´c and Sima’an, 2014), BLEND (Ma et al., 2017), and RUSE (Shimanaka et al., 2018).”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “the creation of many learned metrics, some of which use regression or deep learning (Stanojevic and Sima’an, 2014; Ma et al., 2017; Shimanaka et al., 2018; Chen et al., 2017; Mathur et al., 2019).”
MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distance (2019, background): “Baselines We select multiple strong baselines for each task for comparison: SentBLEU, ME-TEOR++ (Guo et al., 2018), and a supervised metric RUSE for machine translation; ROUGE-1 and ROUGE-2 and a supervised metric S 3 best (Peyrard et al., 2017) for text summarization; BLEU and METEOR for dialogue r”
Evaluation of Text Generation: A Survey (2020, background): “Shimanaka et al. (2018) propose a segment-level machine translation evaluation metric named RUSE.”
COMET: A Neural Framework for MT Evaluation (2020, methodology): “Learnable metrics (Shimanaka et al., 2018; Mathur et al., 2019) attempt to directly optimize the correlation with human judgments, and have recently shown promising results.”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, methodology): “, 2015), RUSE (Shimanaka et al., 2018), WMDO (Chow et al.”
MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers (2021, background): “More recent reference-based metrics are capable of comparisons in a high dimensional space [53, 63, 51, 9], thereby capturing distributional semantics beyond superficial n-gram statistics.”
FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metrics for Automatic Text Generation (2021, methodology): “RUSE (Shimanaka et al., 2018) evaluates machine translation with a neural regressor based on universal sentence embeddings (e.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “ntence�, and �,�,��′are parameters to be learnt. The metric is trained on the Direct Assessment human evaluation data that is collected for WMT 2016 [12]. RUSE (Regressor Using Sentence Enbeddings [143]): RUSE is a MultiLayer Perceptron (MLP) based regression model that combines three pre-trained”
UNION: An Unreferenced Metric for Evaluating Open-ended Story Generation (2020, background): “RUSE trained a regression model to score a reference-candidate pair using their sentence embeddings.”
SEScore2: Retrieval Augmented Pretraining for Text Generation Evaluation (2022, methodology): “Inspired by the prior approach (Shimanaka et al., 2018), we extract two features between sentence embeddings x 0 and x n : 1) element-wise sentence product and 2) element-wise sentence difference.”
Measuring Similarity of Opinion-bearing Sentences (2021, background): “For example, machine translation metrics learn to score machine translations against humans translations from previous machine translation datasets with human annotations (Shimanaka et al., 2018; Mathur et al., 2019).”
Not All Errors are Equal: Learning Text Generation Metrics using Stratified Error Synthesis (2022, unknown): “Then we use the approach proposed by RUSE (Shimanaka et al., 2018) to extract the two features: 1) Element-wise synthesized and reference sentence product.”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “On the other hand, fine-tuned metrics such as RUSE (Shimanaka et al., 2018), BLEURT (Sellam et al.”