Towards a Better Metric for Evaluating Question Generation Systems

By Preksha Nema, Mitesh M. Khapra. 2018

122 citations recorded by the source corpus.

There has always been criticism for using n-gram based similarity metrics, such as BLEU, NIST, etc, for evaluating the performance of NLG systems. However, these metrics continue to remain popular and are recently being used for evaluating the performance of systems which automatically generate questions from documents, knowledge graphs, images, etc. Given the rising interest in such automatic question generation (AQG) systems, it is important to objectively examine whether these metrics are suitable for this task. In particular, it is important to verify whether such metrics used for evaluating AQG systems focus on answerability of the generated question by preferring questions which contain all relevant information such as question type (Wh-types), entities, relations, etc. In this work, we show that current automatic evaluation metrics based on n-gram similarity do not always correlate well with human judgments about answerability of a question. To alleviate this problem and as a first step towards better evaluation metrics for AQG, we introduce a scoring function to capture answerability and show that when this scoring function is integrated with existing metrics, they correlate significantly better with human judgments. The scripts and data developed as a part of this work are made publicly available.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Following [60], we evaluate generations using the SQuAD-tuned Q-BLEU-1 metric [37].”
Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, methodology): “Second, we apply the new Q-BLEU1 metric proposed by (Nema and Khapra, 2018).”
Evaluation of Text Generation: A Survey (2020, methodology): “Although mainly designed for evaluating single- or multi-document summarization, it has also been used for evaluating short text generation, such as machine translation (Lin & Och, 2004), image captioning (Cui et al., 2018), and question generation (Nema & Khapra, 2018; Dong et al., 2019).”
Perturbation CheckLists for Evaluating NLG Evaluation Metrics (2021, methodology): “For the tasks of QG, AS, IC and D2T we ﬁnd that task-agnostic metrics such as BERTScore, Moverscore and BLEURT are consistently among the top performing metrics (i.e., their correlation scores are either the best or close to the best scores for a given task and criteria) along with the task-speciﬁc ”
Towards a Decomposable Metric for Explainable Evaluation of Text Generation from AMR (2020, background): “Yet, such metrics suffer from several well-known issues (Novikova et al., 2017; Nema and Khapra, 2018; Sai et al., 2020).”