Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering

By Shiyue Zhang, Mohit Bansal. 2019

143 citations recorded by the source corpus.

Text-based Question Generation (QG) aims at generating natural and relevant questions that can be answered by a given answer in some context. Existing QG models suffer from a “semantic drift” problem, i.e., the semantics of the model-generated question drifts away from the given context and answer. In this paper, we first propose two semantics-enhanced rewards obtained from downstream question paraphrasing and question answering tasks to regularize the QG model to generate semantically valid questions. Second, since the traditional evaluation metrics (e.g., BLEU) often fall short in evaluating the quality of generated questions, we propose a QA-based evaluation method which measures the QG model’s ability to mimic human annotators in generating QA training data. Experiments show that our method achieves the new state-of-the-art performance w.r.t. traditional metrics, and also performs best on our QA-based evaluation metrics. Further, we investigate how to use our QG model to augment QA datasets and enable semi-supervised QA. We propose two ways to generate synthetic QA pairs: generate new questions from existing articles or collect QA pairs from new articles. We also propose two empirically effective strategies, a data filter and mixing mini-batch training, to properly use the QG-generated data for QA. Experiments show that our method improves over both BiDAF and BERT QA baselines, even without introducing new articles.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Following [60], we evaluate generations using the SQuAD-tuned Q-BLEU-1 metric [37].”
Unified Language Model Pre-training for Natural Language Understanding and Generation (2019, methodology): “SemQG [49] uses two semantics-enhanced rewards to regularize the generation.”
ColdGANs: Taming Language GANs with Cautious Sampling Strategies (2020, neutral): “(CNN/DM) #params BLEU-1 BLEU-4 ROUGE-1 ROUGE-L BLEU-4 SemQG [56] 18.”
UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Training (2020, methodology): “Among the compared results, UNILM (Dong et al., 2019) is based on pretrained models, while the other three methods are sequenceto-sequence models enhanced with manual features (Du & Cardie, 2018), gated self-attention (Zhao et al., 2018), and reinforcement learning (Zhang & Bansal, 2019).”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, methodology): “Hence, we retain the most informative data samples from the synthetic pool by using Question Answering Probability (Zhang and Bansal, 2019) to measure accuracy by answerability.”
Quiz-Style Question Generation for News Stories (2021, unknown): “There have also been various extensions, including copy or pointer mechanisms to overcome out-of-vocabulary issues [11, 44], mechanisms for handling paragraph-length inputs [50, 51], and more [19, 33].”
Evaluating Factuality in Generation with Dependency-level Entailment (2020, methodology): “Besides the data generation procedure we proposed, there are other ways to synthetically generate noisy annotations for premise-hypothesis pairs (Zhang et al., 2019).”
A Survey on Deep Semi-Supervised Learning (2021, unknown): “Semantic role labeling CoNLL-2009 [127], CoNLL-2013 [128] [129], [130], [131] Question answering SQuAD [132], TriviaQA [133] [134], [135], [136]”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, methodology): “We see largest improvements with using QAP as the filter.”
ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training (2020, methodology): “Following Dong et al. (2019), we compare our model against the following models: CorefNQG (Du & Cardie, 2018) which employs a feature-rich encoder based on Seq2Seq model; MP-GSN (Zhao et al., 2018) which incorporates a gated self-attention encoder with maxout pointer; SemQG (Zhang & Bansal, 2019) wh”
An Empirical Survey of Data Augmentation for Limited Data Learning in NLP (2021, methodology): “After training the model to generate the original text given the label, the model can generate new text (Anaby-Tavor et al., 2020; Zhang and Bansal, 2019; Kumar et al., 2020; Yang et al., 2020).”