YiSi - a Unified Semantic MT Quality Evaluation and Estimation Metric for Languages with Different Levels of Available Resources

By Chi-kiu (羅致翹) Lo. 2019

154 citations recorded by the source corpus.

We present YiSi, a unified automatic semantic machine translation quality evaluation and estimation metric for languages with different levels of available resources. Underneath the interface with different language resources settings, YiSi uses the same representation for the two sentences in assessment. Besides, we show significant improvement in the correlation of YiSi-1’s scores with human judgment is made by using contextual embeddings in multilingual BERT–Bidirectional Encoder Representations from Transformers to evaluate lexical semantic similarity. YiSi is open source and publicly available.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Machine Translation

Citations in the corpus

DART: Open-Domain Structured Data Record to Text Generation (2020, neutral): “, 2020a), YiSi-1 (Lo, 2019), WMD (Kusner et al.”
Evaluation of Text Generation: A Survey (2020, methodology): “YISI Inspired by the MEANT score, YISI3 (Lo, 2019) is proposed to evaluate the accuracy of machine translation model outputs.”
COMET: A Neural Framework for MT Evaluation (2020, neutral): “, 2016), BLEU2VEC (Tättar and Fishel, 2017), YISI-1 (Lo, 2019), MOVERSCORE (Zhao et al.”
Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics (2020, methodology): “Best metrics across language pairs • Y I S I -1 (Lo, 2019) computes the semantic similarity of phrases in the MT output with the reference, using contextual word embeddings (BERT: Devlin et al. (2019)).”
Multi-Source Transformer for Kazakh-Russian-English Neural Machine Translation (2019, methodology): “Since the Paracrawl portion of the bilingual data is very noisy, before training our ﬁnal submitted system we ran our parallel corpus ﬁltering pipeline (Lo et al., 2018) with YiSi-2 as the scoring function (instead of MT + YiSi-1) and trimmed the size of the Paracrawl portion from 12M sentence pairs”
CLIPScore: A Reference-free Evaluation Metric for Image Captioning (2021, background): “But recent work (Lo, 2019; Yankovskaya et al., 2019; Zhao et al., 2020) has improved correlation with human judgment not by gathering more monolingual references, but instead by utilizing cross-lingual representations learned by large-scale, pre-trained, multilingual models e.”
Dialect-robust Evaluation of Generated Text (2022, background): “Among previous works, BLEURT (Pu et al., 2021) and YISI (Lo, 2018) are trained to measure the semantic similarity between candidate and reference within the same language.”
BERT: A Review of Applications in Natural Language Processing and Understanding (2021, methodology): “Variations of classical metrics [45]using BERT and showing improved correlation with human estimates are presented.”
Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factors (2021, unknown): “, 2020), Yisi-2 (Lo, 2019), KoBE (Gekhman et al.”
Toxicity in Multilingual Machine Translation at Scale (2022, methodology): “Automatic measures compare the translation output to one or more human references, e.g., Papineni et al. (2002); Popovi´c (2015a); Lo (2019); Rei et al. (2020); Sellam et al. (2020); Freitag et al. (2021), or pre-trained embeddings, e.g., Lo (2019); Yankovskaya et al. (2019).”
Results of WMT22 Metrics Shared Task: Stop Using BLEU – Neural Metrics Are Better and More Robust (2022, methodology): “Y I S I -1 (Lo, 2019) is a MT evaluation metric that measures the semantic similarity between a machine translation and human references by aggregating the IDF-weighted lexical semantic similarities based on the contextual embeddings extracted from pre-trained language models (e.g. RoBERTa,…”
Reward Optimization for Neural Machine Translation with Learned Metrics (2021, background): “Yisi-1 (kiu Lo, 2019) considers the distributional lexical similarity.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, background): “…tokenizer chrF [100] * * – characTER [138] tokenizer EED [123] tokenizer Vector Extrema [42 * tokenizer, BOW vectors, pretrained embeddings YiSi [74] tokenizer, BOW vectors, pretrained embeddings BERTr [84] * tokenizer, BERT embeddings BERTscore [144] * * tokenizer, BERT embeddings MoverScore…”
Results of the WMT21 Metrics Shared Task: Evaluating Metrics with Expert-based Human Evaluations on TED and News Domain (2021, methodology): “We use Kendall’s tau-like correlation, typically used for DA RR (Bojar et al., 2017; Ma et al., 2018, 2019; Mathur et al., 2020b), for evaluating the primary submissions on the challenge sets.”
NUBIA: NeUral Based Interchangeability Assessor for Text Generation (2020, background): “YiSi (Lo, 2019) also deﬁnes a distance metric among reference and hypothesis sentences based on multilingual BERT embeddings and word frequency weightings.”