Training Question Answering Models from Synthetic Data

By Raul Puri, Ryan Spring, M. Patwary, Mohammad Shoeybi, Bryan Catanzaro. 2020

179 citations recorded by the source corpus.

Question and answer generation is a data augmentation method that aims to improve question answering (QA) models given the limited amount of human labeled data. However, a considerable gap remains between synthetic and human-generated question-answer pairs. This work aims to narrow this gap by taking advantage of large language models and explores several factors such as model size, quality of pretrained models, scale of data synthesized, and algorithmic choices. On the SQuAD1.1 question answering task, we achieve higher accuracy using solely synthetic questions and answers than when using the SQuAD1.1 training set questions alone. Removing access to real Wikipedia data, we synthesize questions and answers from a synthetic corpus generated by an 8.3 billion parameter GPT-2 model. With no access to human supervision and only access to other models, we are able to train state of the art question answering networks on entirely model-generated data that achieve 88.4 Exact Match (EM) and 93.9 F1 score on the SQuAD1.1 dev set. We further apply our methodology to SQuAD2.0 and show a 2.8 absolute gain on EM score compared to prior work using synthetic data.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “…(He et al., 2020) or on variations of consistency (Xie et al., 2019; Sohn et al., 2020) for a given task–such as round-trip consistency of question generation and answer prediction (Alberti et al., 2019; Puri et al., 2020) for QA or between source and targets in summarization (Fabbri et al., 2021).”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “A long line of prior work has trained generative models for question answering (Du et al., 2017; Du and Cardie, 2018; Zhao et al., 2018; Lewis and Fan, 2019; Alberti et al., 2019; Puri et al., 2020; Yang et al., 2020; Bartolo et al., 2021; Lewis et al., 2021).”
BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation (2022, methodology): “Recently, generative language models have been used to synthesize examples for various NLP tasks (Kumar et al., 2020; Anaby-Tavor et al., 2020; Puri et al., 2020; Yang et al., 2020).”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, background): “Large pretrained models like GPT2 (Radford et al., 2018) can be finetuned to generate sentences from narrow data distributions, and it has recently been leveraged to augment datasets for text classification (Anaby-Tavor et al., 2020) and question answering (Puri et al., 2020; Yang et al., 2020).”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, methodology): “We also note that using only synthetic generated data, we can achieve good performance on D SQuAD consistent with the ﬁndings of Puri et al. (2020), and outperform the model trained on the human-written SQuAD data on D BERT (+0.6F 1 ) and D RoBERTa (+6.6F 1 ).”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, background): “novel examples for text classification and reading comprehension (Anaby-Tavor et al., 2020; Kumar et al., 2020; Puri et al., 2020).”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, background): “, 2020), and question generation models (Alberti et al., 2019; Puri et al., 2020; Bartolo et al., 2021).”
Leveraging QA Datasets to Improve Generative Data Augmentation (2022, methodology): “Puri et al. (2020) use GLMs to synthesize questions and answers and improve performance on question answering.”
SynthBio: A Case Study in Human-AI Collaborative Curation of Text Datasets (2021, background): “[40] achieve state-of-the-art results on a question answering model trained only on synthetic data generated by GPT-2.”
ZeroGen: Efficient Zero-shot Learning via Dataset Generation (2022, background): “Early efforts achieve this goal with fine-tuned generative models (Anaby-Tavor et al., 2020; Puri et al., 2020; Kumar et al., 2020; Lee et al., 2021).”
All You May Need for VQA are Image Captions (2022, background): “…language models (Narayan et al., 2020) and as a component in solutions for other textual tasks, such as question answering (Al-berti et al., 2019; Puri et al., 2020), information retrieval (Mass et al., 2020; Gaur et al., 2021) and generation evaluation (Durmus et al., 2020; Wang et al., 2020;…”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “The model receives the passage as context and must generate the answer text, then a special separator token, then the question; this approach is simpler than prior approaches that use separate models for answer and question generation (Lewis and Fan, 2019; Alberti et al., 2019; Puri et al., 2020), a”
Relation-Guided Pre-Training for Open-Domain Question Answering (2021, unknown): “Some follow-up works also utilize template (Fabbri et al., 2020) and pre-trained language model (Puri et al., 2020) over masked cloze-style questions for more human-readable questions.”
GraDA: Graph Generative Data Augmentation for Commonsense Reasoning (2022, background): “Large pretrained models like GPT2 (Radford et al., 2018) can be finetuned to generate sentences from narrow data distributions, and it has recently been leveraged to augment datasets for text classification (Anaby-Tavor et al., 2020) and question answering (Puri et al., 2020; Yang et al., 2020).”
STraTA: Self-Training with Task Augmentation for Better Few-shot Learning (2021, background): “…(Rad-ford et al., 2019) and T5 (Raffel et al., 2020), to generate synthetic training data for different tasks, including text classiﬁcation (Anaby-Tavor et al., 2020; Lee et al., 2021; Schick and Schütze, 2021), question answering (Puri et al., 2020), and commonsense reasoning (Yang et al., 2020).”