Unsupervised Question Answering by Cloze Translation

By Patrick Lewis, Ludovic Denoyer, Sebastian Riedel. 2019

142 citations recorded by the source corpus.

Obtaining training data for Question Answering (QA) is time-consuming and resource-intensive, and existing QA datasets are only available for limited domains and languages. In this work, we explore to what extent high quality training data is actually required for Extractive QA, and investigate the possibility of unsupervised Extractive QA. We approach this problem by first learning to generate context, question and answer triples in an unsupervised manner, which we then use to synthesize Extractive QA training data automatically. To generate such triples, we first sample random context paragraphs from a large corpus of documents and then random noun phrases or Named Entity mentions from these paragraphs as answers. Next we convert answers in context to “fill-in-the-blank” cloze questions and finally translate them into natural questions. We propose and compare various unsupervised ways to perform cloze-to-natural question translation, including training an unsupervised NMT model using non-aligned corpora of natural questions and cloze questions as well as a rule-based approach. We find that modern QA models can learn to answer human questions surprisingly well using only synthetic training data. We demonstrate that, without using the SQuAD training data at all, our approach achieves 56.4 F1 on SQuAD v1 (64.5 F1 when the answer is a Named Entity mention), outperforming early supervised models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

CTRL: A Conditional Transformer Language Model for Controllable Generation (2019, background): “For important skills, the inclusion of supervised data or task-speciﬁc data generated through unsupervised means (Artetxe et al., 2017; Lewis et al., 2019) can lead to obvious improvements.”
How Context Affects Language Models' Factual Predictions (2020, methodology): “Radford et al. [2019] and Lewis et al. [2019b] demonstrate non-trivial performance on CoQA [Reddy et al., 2019] and SQuAD [Rajpurkar et al., 2016] respectively, and [Yadav et al., 2019] achieve SoTA results using an unsupervised method for multi-choice QA on ARC [Clark et al., 2018].”
MLQA: Evaluating Cross-lingual Extractive Question Answering (2019, background): “Other potential areas to explore involve training datasets other than English, such as CMRC (Cui et al., 2018), or using unsupervised QA techniques to assist transfer (Lewis et al., 2019).”
Unsupervised Question Decomposition for Question Answering (2020, methodology): “Supervised decomposition methods are limited by the amount of available human annotation, but our unsupervised method faces no such limitation, similar to unsupervised QA (Lewis et al., 2019).”
Eliciting Knowledge from Language Models Using Automatically Generated Prompts (2020, background): “, 2020), question answering (Lewis et al., 2019), fact recall (Petroni et al.”
Improving Question Answering Model Robustness with Synthetic Adversarial Data Generation (2021, methodology): “Baseline Systems We investigate three baseline systems; noun phrases and named entities following Lewis et al. (2019), as well as an extended part-of-speech tagger incorporating named entities, adjectives, noun phrases, numbers, distinct proper nouns, and clauses.”
Can Small and Synthetic Benchmarks Drive Modeling Innovation? A Retrospective Study of Question Answering Modeling Approaches (2021, methodology): “This procedure is inspired by the noisy cloze generation method of Lewis et al. (2019).”
Zero-shot Neural Retrieval via Domain-targeted Synthetic Query Generation (2020, methodology): “Question generation has also been employed to augment QA training sets (Alberti et al., 2019; Lewis et al., 2019), but has never been tested for ad-hoc retrieval nor the zero-shot setting.”
Training Question Answering Models from Synthetic Data (2020, methodology): “Earlier work (Dhingra et al., 2018; Lewis et al., 2019) using named entity and noun phrase answer candidates performed best only on those portions of the data distribution.”
Adversarial Language Games for Advanced Natural Language Intelligence (2019, background): “Specifically, the attacker is instantiated by a neural question generation model (Lewis et al., 2019) that generates questions based on target words and sentences from Wikipedia by unsupervised neural machine translation.”
Question Answering Infused Pre-training of General-Purpose Contextualized Representations (2021, methodology): “To do this, we train on data generated by the Unsupervised QA pipeline of Lewis et al. (2019).”
Teaching Machine Comprehension with Compositional Explanations (2020, background): “Similar to the few-shot evaluation in Lewis et al. (2019), we randomly sample different number of human-annotated instances”
Multi-Stage Pretraining for Low-Resource Domain Adaptation (2020, background): “Golub et al. (2017), Yang et al. (2017), Lewis et al. (2019) and Dhingra et al. (2018) develop approaches to generate natural questions that can aid downstream question answering tasks.”
Relation-Guided Pre-Training for Open-Domain Question Answering (2021, unknown): “QA ) (Lewis et al., 2019) as a baseline.”