BEER: BEtter Evaluation as Ranking

By Miloš Stanojević, Khalil Sima'an. 2014

119 citations recorded by the source corpus.

We present the UvA-ILLC submission of the BEER metric to WMT 14 metrics task. BEER is a sentence level metric that can incorporate a large number of features combined in a linear model. Novel contributions are (1) efficient tuning of a large number of features for maximizing correlation with human system ranking, and (2) novel features that give smoother sentence level scores.

Subfields: Artificial Intelligence, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

Citations in the corpus

Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Embed-ding based approaches (Stanojevic and Sima’an, 2014) may be language agnostic and help address these issues.”
BERTScore: Evaluating Text Generation with BERT (2019, methodology): “We also compare our correlations with state-of-the-art metrics, including METEOR++ (Guo et al., 2018), CHR F++ (Popovi´c, 2017), BEER (Stano-jevi´c and Sima’an, 2014), BLEND (Ma et al., 2017), and RUSE (Shimanaka et al., 2018).”
BLEURT: Learning Robust Metrics for Text Generation (2020, background): “the creation of many learned metrics, some of which use regression or deep learning (Stanojevic and Sima’an, 2014; Ma et al., 2017; Shimanaka et al., 2018; Chen et al., 2017; Mathur et al., 2019).”
Nematus: a Toolkit for Neural Machine Translation (2017, neutral): “Various MT metrics are supported as loss function, including smoothed sentence-level BLEU (Chen and Cherry, 2014), METEOR (Denkowski and Lavie, 2011), BEER (Stanojevic and Sima’an, 2014), and any interpolation of implemented metrics.”
Fitting Sentence Level Translation Evaluation with Many Dense Features (2014, methodology): “BEER participated on WMT14 evaluation metrics task where it was the highest scoring sentence level evaluation metric on average over all language pairs (Stanojević and Sima’an, 2014) https://github.”
BARTScore: Evaluating Generated Text as Text Generation (2021, methodology): “Examples include COMET [57] and BEER [65], where COMET focuses the machine translation task and relies on human judgments to tune parameters in ranking or regression layers, and BEER combines many simple features in a tunable linear model of MT evaluation metrics.”
FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, background): “To enable performance analyses from different perspectives, such similarity is measured in terms of multiple automatic metrics: case-sensitive/insensitive BLEU (Papineni et al., 2002b), case-sensitive/insensitive TER (Snover et al., 2006a), BEER (Stanojevic and Sima'an, 2014), and CharacTER (Wang et”
InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation (2021, background): “, RUSE (Shimanaka, Kajiwara, and Komachi 2018), BEER (Stanojević and Sima’an 2014), BLEND (Ma et al.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “BEER (BEtter Evaluation as Ranking) [150, 151]: The set of input features used by BEER include precision, recall and F1-score on character n-grams for various n and on word-level unigrams.”
Tilde’s Machine Translation Systems for WMT 2019 (2019, methodology): “We evaluated all MT systems using multiple automatic evaluation metrics including BLEU (Papineni et al., 2002), BEER 2.0 (Stanojevic and Sima’an, 2014), CharacTER (Wang et al., 2016), and TER (Snover et al., 2006).”
The IWSLT 2018 Evaluation Campaign (2018, methodology): “In addition, we calculated the BEER score [13] and the characTER [14].”