Putting the Horse before the Cart: A Generator-Evaluator Framework for Question Generation from Text

By Vishwajeet Kumar, Ganesh Ramakrishnan, Yuan-Fang Li. 2018

44 citations recorded by the source corpus.

Automatic question generation (QG) is a useful yet challenging task in NLP. Recent neural network-based approaches represent the state-of-the-art in this task. In this work, we attempt to strengthen them significantly by adopting a holistic and novel generator-evaluator framework that directly optimizes objectives that reward semantics and structure. The generator is a sequence-to-sequence model that incorporates the structure and semantics of the question being generated. The generator predicts an answer in the passage that the question can pivot on. Employing the copy and coverage mechanisms, it also acknowledges other contextually important (and possibly rare) keywords in the passage that the question needs to conform to, while not redundantly repeating words. The evaluator model evaluates and assigns a reward to each predicted question based on its conformity to the structure of ground-truth questions. We propose two novel QG-specific reward functions for text conformity and answer conformity of the generated question. The evaluator also employs structure-sensitive rewards based on evaluation measures such as BLEU, GLEU, and ROUGE-L, which are suitable for QG. In contrast, most of the previous works only optimize the cross-entropy loss, which can induce inconsistencies between training (objective) and testing (evaluation) measures. Our evaluation shows that our approach significantly outperforms state-of-the-art systems on the widely-used SQuAD benchmark as per both automatic and human evaluation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Mohit Bansal
2. Shiyue Zhang
3. Wei Wu
4. Jiwei Li
5. Fei Wu
6. Duo Chai
7. Jianfeng Gao
8. Qinghong Han

In Computer Vision and Pattern Recognition

1. Mohit Bansal
2. Shiyue Zhang
3. Wei Wu
4. Fei Wu
5. Jiwei Li
6. Vishwajeet Kumar
7. Ganesh Ramakrishnan
8. Luc Van Gool

In Natural Language Processing

1. Mohit Bansal
2. Shiyue Zhang
3. Jiwei Li
4. Wei Wu
5. Nitish Joshi
6. Preethi Jyothi
7. Arijit Mukherjee
8. Fei Wu

Citations in the corpus

• Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, methodology): “To address the exposure bias problem, some works applied reinforcement learning taking evaluation metrics (e.g., BLEU) as rewards (Song et al., 2017; Kumar et al., 2018).”
• Description Based Text Classification with Reinforcement Learning (2020, methodology): “Our work is similar to Yuan et al. (2017) and Kumar et al. (2018) in terms of description generation, in which reinforcement learning is applied for description/query generation.”
• Cross-Lingual Training for Automatic Question Generation (2019, methodology): “Following Kumar et al. (2018a), we measure the quality of questions in terms of syntactic correctness, semantic correctness and relevance.”