Learning Compact Metrics for MT

By Amy Pu, Hyung Won Chung, Ankur P. Parikh, Sebastian Gehrmann, Thibault Sellam. 2021

119 citations recorded by the source corpus.

Recent developments in machine translation and multilingual text generation have led researchers to adopt trained metrics such as COMET or BLEURT, which treat evaluation as a regression problem and use representations from multilingual pre-trained models such as XLM-RoBERTa or mBERT. Yet studies on related tasks suggest that these models are most efficient when they are large, which is costly and impractical for evaluation. We investigate the trade-off between multilinguality and model capacity with RemBERT, a state-of-the-art multilingual language model, using data from the WMT Metrics Shared Task. We present a series of experiments which show that model size is indeed a bottleneck for cross-lingual transfer, then demonstrate how distillation can help addressing this bottleneck, by leveraging synthetic data generation and transferring knowledge from one teacher to multiple students trained on related languages. Our method yields up to 10.5% improvement over vanilla fine-tuning and reaches 92.6% of RemBERT’s performance using only a third of its parameters.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

PaLM: Scaling Language Modeling with Pathways (2022, methodology): “Metrics Following the suggestions by Gehrmann et al. (2022), we report ROUGE-2, ROUGE-L (Lin, 2004), and BLEURT-20 (Pu et al., 2021).15 For brevity, the main text of this section focuses on the F-measure of ROUGE-2 while the remaining results, including precision and recall numbers for ROUGE, are in”
Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text (2022, methodology): “Time Travel Dataset shifts are when the joint distribution of inputs and outputs differs between the development of a model and its deployment (or in our case test its test setup) (Quiñonero-Candela et al., 2009). One of the suggestions by Søgaard et al. (2021) is to simulate dataset shifts to simul”
GLM-130B: An Open Bilingual Pre-trained Model (2022, methodology): “13We use ROUGE implementation at https://github.com/google-research/google-research/tree/master/rouge and BLEURT-20 implementation at https://github.com/google-research/google-research/tree/master/rouge, whose checkpoint is available at https://storage.googleapis.com/bleurt-oss-21/BLEURT-20.zip (E2E”
Time Travel in LLMs: Tracing Data Contamination in Large Language Models (2023, methodology): “We highlight that our BLEURT score computations use the most recent checkpoint provided, i.e., BLEURT-20 (Pu et al. 2021).”
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “We used the B LEURT 20 checkpoint (Pu et al., 2021) which was trained on top of RemBERT us-ing DA from previous shared tasks ranging 2015 to 2019 and additional synthetic data created from Wikipedia articles.”
COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “As for baselines we used lexical metrics such as CHR F (Popovi´c, 2015) and B LEU (Papineni et al., 2002) and three state-of-the-art metrics: B LEURT (Sellam et al., 2020; Pu et al., 2021), C OMET -20 (Rei et al., 2020b) and C OMET -21 (Rei et al., 2021) 3 .”
Intriguing Properties of Compression on Multilingual Models (2022, background): “While most compression techniques have minimal impact on aggregate performance numbers (Gale et al., 2019; Li et al., 2020; Hou et al., 2020; Chen et al., 2021; Bai et al., 2020; ab Tessera et al., 2021), the impact on individual sub-populations in the data, such as low-resource languages, can be…”
TaTa: A Multilingual Table-to-Text Dataset for African Languages (2022, methodology): “(1) Reference-Based : We assess {P,R,F}-score variants of ROUGE-{1,2,L} (Lin, 2004) as n-gram based metric, CHR F (Popovi´c, 2015) with a maximum n-gram order of 6 and β = 2 as character-based metric, and BLEURT-20 (Sellam et al., 2020; Pu et al., 2021) as learned metric.”
xcomet: Transparent Machine Translation Evaluation through Fine-grained Error Detection (2023, methodology): “(9)This technique was initially introduced in BLEURT-20 (Pu et al., 2021).”
TRUE: Re-evaluating Factual Consistency Evaluation (2022, unknown): “We use the recommended BLEURT-20 checkpoint (Pu et al., 2021).”
Evaluating Attribution in Dialogue Systems: The BEGIN Benchmark (2021, methodology): “For BLEURT, we use the recommended BLEURT-20 checkpoint (Pu et al., 2021).”