Synthetic QA Corpora Generation with Roundtrip Consistency

By Chris Alberti, D. Andor, Emily Pitler, Jacob Devlin, Michael Collins. 2019

275 citations recorded by the source corpus.

We introduce a novel method of generating synthetic question answering corpora by combining models of question generation and answer extraction, and by filtering the results to ensure roundtrip consistency. By pretraining on the resulting corpora we obtain significant improvements on SQuAD2 and NQ, establishing a new state-of-the-art on the latter. Our synthetic data generation models, for both question generation and answer extraction, can be fully reproduced by finetuning a publicly available BERT model on the extractive subsets of SQuAD2 and NQ. We also describe a more powerful variant that does full sequence-to-sequence pretraining for question generation, obtaining exact match and F1 at less than 0.1% and 0.4% from human performance on SQuAD2.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Addressing Semantic Drift in Question Generation for Semi-Supervised Question Answering (2019, neutral): “Similar methods have also been applied in some very recent concurrent works (Dong et al., 2019; Alberti et al., 2019) on SQuADv2.”
On Faithfulness and Factuality in Abstractive Summarization (2020, methodology): “Given an abstractive summary, we used the round-trip consistency method of Alberti et al. (2019), which combines question generation and answer extraction models to generate synthetic question-answer pairs.”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, background): “The ﬁeld has made rapid advances in recent years, even exceeding human performance in some settings (Devlin et al., 2019; Alberti et al., 2019).”
Few-shot Mining of Naturally Occurring Inputs and Outputs (2022, background): “…(He et al., 2020) or on variations of consistency (Xie et al., 2019; Sohn et al., 2020) for a given task–such as round-trip consistency of question generation and answer prediction (Alberti et al., 2019; Puri et al., 2020) for QA or between source and targets in summarization (Fabbri et al., 2021).”
Models in the Loop: Aiding Crowdworkers with Generative Annotation Assistants (2021, background): “A long line of prior work has trained generative models for question answering (Du et al., 2017; Du and Cardie, 2018; Zhao et al., 2018; Lewis and Fan, 2019; Alberti et al., 2019; Puri et al., 2020; Yang et al., 2020; Bartolo et al., 2021; Lewis et al., 2021).”
Just Ask: Learning to Answer Questions from Millions of Narrated Videos (2020, positive): “We follow [4, 13, 52] and use the weights available at [3] trained for answer span extraction and answer-aware question generation, respectively, on SQuADv1 [63].”
Measuring and Improving Consistency in Pretrained Language Models (2021, background): “Alberti et al. (2019) generate new questions conditioned on context and answer from a labeled dataset and by ﬁltering answers that do not provide a consistent result with the original answer.”
PAQ: 65 Million Probably-Asked Questions and What You Can Do With Them (2021, methodology): “Following recent best practices(Alberti et al., 2019; Yang et al., 2019), we then ﬁne-tune on the training QA-pairs only, using validation Exact Match score for early stopping (Rajpurkar et al., 2018).”
Domain-matched Pre-training Tasks for Dense Retrieval (2021, background): “This leads to higher quality, less ambiguous open-domain-style questions than can be achieved standard consistency filtering with machine-comprehension models (Alberti et al., 2019).”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, methodology): “We ﬁne-tune a RoBERTa Large span extraction model as investigated in previous work (Alberti et al., 2019; Lewis and Fan, 2019).”
G-DAug: Generative Data Augmentation for Commonsense Reasoning (2020, background): “Previous work has also presented meth-ods to generate questions for reading comprehension (Heilman and Smith, 2010; Rus et al., 2011; Alberti et al., 2019; Puri et al., 2020a”
Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills (2021, background): “, 2020), and question generation models (Alberti et al., 2019; Puri et al., 2020; Bartolo et al., 2021).”
RikiNet: Reading Wikipedia Pages for Natural Question Answering (2020, methodology): “On the NQ test set, our single model obtains 74.3 F1 scores on the long-answer task (LA) and 57.9 F1 scores on the short-answer task (SA) compared to the published best single model (Alberti et al., 2019a) results of 66.8 F1 on LA and 53.9 F1 on SA.”
A Logic-Driven Framework for Consistency of Neural Models (2019, background): “Alberti et al. (2019) imposed answer consistency over generated questions for machine comprehension.”
Span Selection Pre-training for Question Answering (2019, methodology): “Experiments on the NQ dataset use the strategies and model described by Alberti et al. (2019b) to fine tune a BERTLARGE model with a final layer for answerability prediction as well as sequence start/end prediction.”
Dense Text Retrieval Based on Pretrained Language Models: A Survey (2022, background): “[227] propose a pipeline approach to constructing question-answer pairs from large text corpora.”
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (2022, methodology): “Question generation for QA Automatic question generation trained with supervision has been explored in QA for data augmentation (Alberti et al., 2019; Dong et al., 2019; Sultan et al., 2020), or as a way to enrich knowledge bases for QA-pair retriever models (Lewis et al., 2021).”