Automatic Evaluation of Translation Quality for Distant Language Pairs

By Hideki Isozaki, Tsutomu Hirao, Kevin Duh, Katsuhito Sudoh, Hajime Tsukada. Conference on Empirical Methods in Natural Language Processing, 2010

376 citations recorded by the source corpus.

View Paper: https://www.semanticscholar.org/paper/9f8338120c1f1a0bf7b3eabdd6cf0c22cc0b238c

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Chunk-based Decoder for Neural Machine Translation (2017, methodology): “Evaluation Following the WAT ’16 evaluation procedure, we used BLEU (Papineni et al., 2002) and RIBES (Isozaki et al., 2010) to evaluate our models.”
Insertion-based Decoding with Automatically Inferred Generation Order (2019, unknown): “We use four evaluation metrics including BLEU (Papineni et al., 2002), Ribes (Isozaki et al., 2010), Meteor (Banerjee and Lavie, 2005) and TER (Snover et al., 2006) to avoid using a single metric that might be in favor of a particular generation order.”
compare-mt: A Tool for Holistic Comparison of Language Generation Systems (2019, methodology): “…or too-short sentences) or alternative evaluation metrics such as [ 20 , 30 ) [ 30 , 40 ) [ 40 , 50 ) [ 50 , 60 chrF (Popovi´c, 2015) and RIBES (Isozaki et al., 2010). compare-mt also has an extensible Scorer class, which will be used to expand the metrics supported by compare-mt in the…”
Overview of the 5th Workshop on Asian Translation (2019, methodology): “RIBES scores were calculated using RIBES.py version 1.02.4.21 AMFM scores were calculated using scripts created by the technical collaborators listed in the WAT2019 web page.22 All scores for each task were calculated using the corresponding reference translations.”
BERTScore: Evaluating Text Generation with BERT (2019, background): “Finally, n -gram models fail to capture distant dependencies and penalize semantically-critical ordering changes (Isozaki et al., 2010).”
Reward Optimization for Neural Machine Translation with Learned Metrics (2021, methodology): “Until recently, automatic MT evaluation relies on matching-based metrics such as BLEU (Papineni et al., 2002), Meteor (Banerjee and Lavie, 2005), and RIBES (Isozaki et al., 2010).”
EDITOR: An Edit-Based Transformer with Repositioning for Neural Machine Translation with Soft Lexical Constraints (2020, methodology): “Results conﬁrm that enforcing hard constraints increases CPR but degrades translation quality compared to the same model using soft constraints: for LevT, it degrades BLEU by 2.2–3.9 and RIBES by 5.0–6.6.”
Syntax-based Rewriting for Simultaneous Machine Translation (2015, methodology): “This result is less clear, as MT systems are optimized for BLEU and RIBES penalizes word reordering, making it difﬁcult to compare systems that intentionally change word order.”
Similar Southeast Asian Languages: Corpus-Based Case Study on Thai-Laotian and Malay-Indonesian (2016, background): “One is the RIBES meets the Kendall’d τ quite well, to show that reordering is not a serious problem in the translation.”
Adaptation Data Selection using Neural Language Models: Experiments in Machine Translation (2013, background): “, 2002), RIBES (Isozaki et al., 2010), and TER (Snover et al.”
Fitting Sentence Level Translation Evaluation with Many Dense Features (2014, methodology): “Kendall score was regularly shown to have high correlation with human judgment on distant language pairs (Isozaki et al., 2010; Birch and Osborne, 2010).”
Neural Machine Translation with Source-Side Latent Graph Parsing (2017, methodology): “We used BLEU (Papineni et al., 2002), RIBES (Isozaki et al., 2010), and perplexity scores as our evaluation metrics.”
BEER: BEtter Evaluation as Ranking (2014, methodology): “To evaluate word order we follow (Isozaki et al., 2010; Birch and Osborne, 2010) in representing reordering as a permutation and then measuring the distance to the ideal monotone permutation.”
Evaluation of Text Generation: A Survey (2020, methodology): “Rank-based Intuitive Bilingual Evaluation Score (RIBES) (Isozaki et al., 2010) is another un-trained automatic evaluation metric for machine translation.”
Understanding Knowledge Distillation in Non-autoregressive Machine Translation (2019, background): “Notably, this is similar to the one-to-many translation setting in Johnson et al. (2017) but in our case we do not have an explicit signal (e.”
VizSeq: a visual analysis toolkit for text generation tasks (2019, methodology): “N-gram-based metrics To the extent of our knowledge, VizSeq has the best coverage of common n-gram-based metrics, including BLEU (Papineni et al., 2002), NIST (Doddington, 2002), METEOR (Banerjee and Lavie, 2005), TER (Snover et al., 2006), RIBES (Isozaki et al., 2010), chrF (Popović, 2015) and GLE”
Controlling the Voice of a Sentence in Japanese-to-English Neural Machine Translation (2016, background): “The BLEU, RIBES (Isozaki et al., 2010), and AMFM (Banchs et al., 2015) As a result, the thz wave can be detected as a current current signal in the <unk>.”
Evaluating MT systems with BEER (2015, methodology): “Previous work by Isozaki et al. (2010) and Birch and Osborne (2010) used this permutation view of word order and applied Kendall (cid:28) for evaluating its distance from ideal (monotone) word order.”
Sequence-to-Dependency Neural Machine Translation (2017, methodology): “5 , where both BLEU and RIBES (Isozaki et al., 2010) are used for evaluation.”