chrF: character n-gram F-score for automatic MT evaluation

By Maja Popovic. 2015

1,908 citations recorded by the source corpus.

We propose the use of character n-gram F-score for automatic evaluation of machine translation output. Character ngrams have already been used as a part of more complex metrics, but their individual potential has not been investigated yet. We report system-level correlations with human rankings for 6-gram F1-score (CHRF) on the WMT12, WMT13 and WMT14 data as well as segment-level correlation for 6gram F1 (CHRF) and F3-scores (CHRF3) on WMT14 data for all available target languages. The results are very promising, especially for the CHRF3 score – for translation from English, this variant showed the highest segment-level correlations outperforming even the best metrics on the WMT14 shared evaluation task.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “N-gram based metrics (Papineni et al., 2002; Dod-dington, 2002; Wang et al., 2016; Popovi´c, 2015) that measure lexical overlap require tokenization which is highly affected by language speciﬁc factors such as alphabet size and morphological complexity.”
Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “In order to compose a homogeneous data set, we first select translations with references of length 20 to 40, then sort the translation pairs by difference in character n-gram F-score (chrF, β = 3) (Popović, 2015) and length, and pick the top 400 translation pairs with the highest difference in chrF ”
Execution-Based Evaluation for Open-Domain Code Generation (2022, neutral): “ChrF is also originally proposed for the machine translation task, but later adopted for some code evaluation works (Evtikhiev et al., 2022).”
BERTScore: Evaluating Text Generation with BERT (2019, background): “CHR F (Popovi´c, 2015) compares character n -grams, n ≤ 6 in the reference and candidate sentences.”
XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization (2020, methodology): “In order to measure the translation quality between the human-translated test data and our pseudo test data, we compute the BLEU score, and the chrF score (Popovi´c, 2015), which is suitable for measuring the translation quality of some languages such as Chinese and Russian.”
Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages (2020, background): “Table 2 reports evaluation results in terms of BLEU evaluated on the benchmark test set from JW300, and human-targeted TER (HTER) (Snover et al., 2006), BLEU (Papineni et al., 2002) and ChrF (Popovi´c, 2015) against human corrected model translations.”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, methodology): “The metrics chrF and chrF+ (Popović, 2015, 2017) are computed using their original Python implementation, see Table 2.”
Evaluation of Text Generation: A Survey (2020, methodology): “One of these metrics is the CHRF, character n-gram F-score, which uses character n-grams instead of word n-grams to compare the machine translation model output with the reference translations (Popović, 2015).”
Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translation (2021, background): “In general, metrics that are not solely based on surface characteristics do somewhat better, though this pattern is not consistent (for example, chrF (Popović, 2015) has a high correlation of 0.”
COMET: A Neural Framework for MT Evaluation (2020, background): “, 2002), METEOR (Lavie and Denkowski, 2009), and CHRF (Popović, 2015) have been widely studied and improved (Koehn et al.”