Some Issues in Automatic Evaluation of English-Hindi MT: More Blues for BLEU

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Towards a Better Metric for Evaluating Question Generation Systems (2018, background): “Though these metrics are very popular and are used for a wide range of NLG tasks including AQG, there has always been criticism for using these metrics (for example, see (Callison-Burch et al., 2006; R et al., 2007; Callison-Burch, 2009)).”
A Comprehensive Assessment of Dialog Evaluation Metrics (2021, background): “While BLEU performs reasonably when evaluating NLG systems, it has issues reﬂecting grammaticality, use of semantically similar words, and meaning preservation (Novikova et al., 2017; Ananthakrishnan et al., 2007; Sulem et al., 2018; Reiter, 2018).”
Unleashing the potential of prompt engineering in Large Language Models: a comprehensive review (2023, background): “For instance, despite numerous types of research highlighting the limited correlation between BLEU and similar metrics with human assessments, their popularity has remained unaltered [102, 103].”
A Survey of Evaluation Metrics Used for NLG Systems (2020, result): “In particular, despite several studies [4, 16, 126, 145] showing that BLEU and similar metrics do not correlate well with human judgements, there was no decline in their popularity as depicted in Figure 1.”