Fitting Sentence Level Translation Evaluation with Many Dense Features

By Miloš Stanojević, Khalil Sima'an. 2014

79 citations recorded by the source corpus.

Sentence level evaluation in MT has turned out far more difficult than corpus level evaluation. Existing sentence level metrics employ a limited set of features, most of which are rather sparse at the sentence level, and their intricate models are rarely trained for ranking. This paper presents a simple linear model exploiting 33 relatively dense features, some of which are novel while others are known but seldom used, and train it under the learning-to-rank framework. We evaluate our metric on the standard WMT12 data showing that it outperforms the strong baseline METEOR. We also analyze the contribution of individual features and the choice of training data, language-pair vs. target-language data, providing new insights into this task.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Reliability and Learnability of Human Bandit Feedback for Sequence-to-Sequence Reinforcement Learning (2018, methodology): “, 2011) and BEER (Stanojević and Sima’an, 2014) to cover a diverse set of automatic measures for translation quality.”
Evaluating the morphological competence of Machine Translation Systems (2017, methodology): “The automatic scores of the systems submitted at WMT’178 are in Table 4 where we report BLEU, BEER (Stanojević and Sima’an, 2014) and CharacTER (Wang et al.”
A Survey of Evaluation Metrics Used for NLG Systems (2020, methodology): “y from a simple Linear Regressor to a complex Deep Neural Network. We now discuss these different metrics sub-categorized by the learnable model. Linear Regression BEER (BEtter Evaluation as Ranking) [150, 151]: The set of input features used by BEER include precision, recall and F1-score on charact”
Evaluating MT systems with BEER (2015, methodology): “…that are presented elsewhere: • evaluation of word order using permutation trees (PETs) (Stanojević and Sima’an, 2014b) • character n-gram matching (Stanojević and Sima’an, 2014a) • a learning-to-rank model (Stanojević and Sima’an, 2014a) • a corpus level score that decomposes to sentence…”
Exploiting Linguistic Resources for Neural Machine Translation Using Multi-task Learning (2017, methodology): “For all experiments, we ﬁrst show the BLEU score, then the BEER score and ﬁnally the characTER.”
Investigating Data Variance in Evaluations of Automatic Machine Translation Metrics (2022, unknown): “For example, in Table 3, the significant ranking of all metrics on 2010 dataset is “1, 6, 2, 3, 3, 3, 8, 7, 9, 10” where chrF, BEER and chrF+ are with the same relative ranking of 3.”
Better Instruction-Following Through Minimum Bayes Risk (2024, unknown): “MBR decoding MBR decoding has been explored in the context of machine translation using a variety of translation metrics such as COMET (Rei et al., 2020) and BLEURT (Sellam et al., 2020), with promising results (Freitag et al., 2022; 2023; Farinhas et al., 2023; Stanojevi´c & Sima’an, 2014).”