chrF++: words helping character n-grams

By Maja Popovic. 2017

654 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Language and Linguistics, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Language and Linguistics

In Evaluation & Benchmarks

Citations in the corpus

Augmented Language Models: a Survey (2023, methodology): “, 2002a), METEOR (Banerjee and Lavie, 2005), and chrF++ (Popović, 2017), or text2graph ones such as F1, Precision, and Recall.”
BERTScore: Evaluating Text Generation with BERT (2019, background): “CHR F++ (Popovi´c, 2017) extends CHR F to include word bigram matching.”
Results of the WMT18 Metrics Shared Task: Both characters and embeddings achieve good performance (2018, methodology): “• chrF The metrics chrF and chrF+ (Popović, 2015; Popović, 2017) are computed using their original Python implementation.”
Results of the WMT19 Metrics Shared Task: Segment-Level and Strong MT Systems Pose Big Challenges (2019, methodology): “The metrics chrF and chrF+ (Popović, 2015, 2017) are computed using their original Python implementation, see Table 2.”
The Flores-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translation (2021, neutral): “Secondly, we verify that spBLEU offers similar results to other metrics that are tokenizerindependent such as chrF++ (Popović, 2017).”
COMET: A Neural Framework for MT Evaluation (2020, background): “, 2002), METEOR (Lavie and Denkowski, 2009), and CHRF (Popović, 2015) have been widely studied and improved (Koehn et al., 2007; Popović, 2017; Denkowski and Lavie, 2011; Guo and Hu, 2019), but, by design, they usually fail to recognize and capture semantic similarity beyond the lexical level.”
Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasing (2020, background): “Overlap can also be measured at the character level (Popović, 2015, 2017) or using edit distance (Snover et al., 2006).”
RUSE: Regressor Using Sentence Embeddings for Automatic Machine Translation Evaluation (2018, neutral): “0 (Lo, 2017), and chrF++ (Popović, 2017).”
Results of the WMT20 Metrics Shared Task (2020, background): “2 CHRF++ CHRF++ (Popović, 2017) includes word unigrams and bigrams in addition to character ngrams.”
Generative Pretraining for Paraphrase Evaluation (2021, methodology): “BLEU, ROUGE, METEOR and chrF++ (Popović, 2017) are widely used n-grambased methods, working at the word, subword or character level.”
Graph-to-Sequence Learning using Gated Graph Neural Networks (2018, methodology): “, 2014), we also report results using sentence-level CHRF++ (Popović, 2017), using the Wilcoxon signed-rank test to check significance.”
A Human Evaluation of AMR-to-English Generation Systems (2020, background): “When evaluated only on sentences used in the human evaluation, only METEOR, CHRF++, and BERTScore preserve this ranking; BLEU ﬁnds the two essentially tied, while TER ﬁnds Zhu slightly better.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, neutral): “l is given �times more weightage than precision. chrF was initially proposed for evaluating MT systems but has been adopted for other tasks such as image captioning and summarization as well. Popovic [127] propose enhanced versions of chrF, which also contain word �-grams in addition to character �-”
Is ChatGPT A Good Translator? A Preliminary Study (2023, methodology): “We adopt the mostly used BLEU score (Papineni et al., 2002) as our primary metric and also report ChrF++ (Popovi´c, 2017) and TER(Snover et al., 2006) in some cases.”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, methodology): “For chrF++ (Popovic, 2017), TER (Snover et al., 2006), BERTScore (Zhang et al., 2020), and BLEURT (Sellam et al., 2020) we use the repository of Castro Ferreira et al. (2020).”
It’s not a Non-Issue: Negation as a Source of Error in Machine Translation (2020, methodology): “Thus, we also work with three automated metrics, in particular BLEU (Post, 2018), chrF++ (Popovi´c, 2017), and METEOR (Denkowski and Lavie, 2011).”
GPT-too: A Language-Model-First Approach for AMR-to-Text Generation (2020, methodology): “We compute chrF++ (Popovi´c, 2017) using both SacreBLEU and the scripts used by authors of the baseline systems.”