Natural Questions: A Benchmark for Question Answering Research
By Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Michael Collins, Ankur P. Parikh, Chris Alberti, D. Epstein, Illia Polosukhin, Jacob Devlin and 9 more. 2019
4,925 citations recorded by the source corpus.
We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Evaluation & Benchmarks
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Mohit Bansal
- Luke S. Zettlemoyer
- Jianfeng Gao
- Percy Liang
- Graham Neubig
- Yejin Choi
- Quoc V. Le
- Noah A. Smith
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Trevor Darrell
- Xiaogang Wang
- Ming-Hsuan Yang
- Jitendra Malik
- Andrew Zisserman
- Yi Yang
In Information Retrieval & Search
- Jimmy J. Lin
- Jamie Callan
- Luyu Gao
- Hannaneh Hajishirzi
- Wen-tau Yih
- Sebastian Riedel
- Gautier Izacard
- Danqi Chen
In Evaluation & Benchmarks
- Mohit Bansal
- Yejin Choi
- Hannaneh Hajishirzi
- Luke S. Zettlemoyer
- Douwe Kiela
- Ashish Sabharwal
- Samuel R. Bowman
- Matt Gardner
Citations in the corpus
- Training Compute-Optimal Large Language Models (2022, neutral): “On the Natural Questions dataset (Kwiatkowski et al., 2019), Chinchilla achieves new closed-book SOTA accuracies: 31.5% 5-shot and 35.5% 64-shot, compared to 21% and 28% respectively, for Gopher .”
- PaLM: Scaling Language Modeling with Pathways (2022, background): “We first study the three different models (8B, 62B and 540B) on 5 different tasks evaluated in Section 6.1: RTE, Natural Questions, Lambada, Story Cloze, and Trivia QA.”
- LLaMA: Open and Efficient Foundation Language Models (2023, methodology): “We compare LLaMA to existing large language models on two closed-book question answering benchmarks: Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al.”
- Llama 2: Open Foundation and Fine-Tuned Chat Models (2023, methodology): “We evaluate the Llama 2 model together with other open-source models on the NaturalQuestions and TriviaQA benchmarks (Table 22).”
- Mistral 7B (2023, neutral): “…Winogrande [21], PIQA [4], SIQA [22], OpenbookQA [19], ARC-Easy, ARC-Challenge [9], CommonsenseQA [24] • World Knowledge (5-shot): NaturalQuestions [16], TriviaQA [15] • Reading Comprehension (0-shot): BoolQ [8], QuAC [7] • Math: GSM8K [10] (8-shot) with maj@8 and MATH [13] (4-shot) with maj@4 •…”
- Mixtral of Experts (2024, neutral): “…Winogrande [26], PIQA [3], SIQA [27], OpenbookQA [22], ARC-Easy, ARC-Challenge [8], CommonsenseQA [30] • World Knowledge (5-shot): NaturalQuestions [20], TriviaQA [19] • Reading Comprehension (0-shot): BoolQ [7], QuAC [5] • Math: GSM8K [9] (8-shot) with maj@8 and MATH [17] (4-shot) with maj@4 •…”
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Such tasks include open-domain question answering [5, 25], fact checking [50], fact completion [42], long-form question answering [12], Wikipedia article generation [32], dialogue [36, 59, 9, 13], translation [16], and language modeling [17, 23].”
- Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, methodology): “On NaturalQuestions, the closed book T5 model obtains 36.6% accuracy with 11B parameters, while our approach obtains 44.1% with 770M parameters plus Wikipedia with BM25 retrieval.”
- REALM: Retrieval-Augmented Language Model Pre-Training (2020, background): “NaturalQuestions-Open The NaturalQuestions dataset (Kwiatkowski et al., 2019) consists of naturally occurring Google queries and their answers.”
- Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “We evaluate our model on the following tasks and datasets included in KILT: question answering: NaturalQuestions (Kwiatkowski et al., 2019), TriviaQA (Joshi et al., 2017) and HotpotQA (Yang et al., 2018); slot filling: Zero Shot RE (Levy et al., 2017) and T-REx (Elsahar et al., 2018); entity…”
- Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “Question Answering We look at Web Questions (Berant et al., 2013), Natural Questions (Kwiatkowski et al., 2019) and TriviaQA (Joshi et al., 2017).”
- How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, methodology): “2 (Kwiatkowski et al., 2019)).”
- Learning and Evaluating General Linguistic Intelligence (2019, background): “…to solve with little in the way of generalization or abstraction ; for instance, ever larger datasets created by crowd-sourcing processes that may not well approximate the natural distributions they are intended to span, although there are some notable counter-examples (Kwiatkowski et al., 2019).”
- Improving Question Answering with External Knowledge (2019, neutral): “In contrast to other types of QA scenarios (Nguyen et al., 2016; Dhingra et al., 2017; Joshi et al., 2017; Dunn et al., 2017; Kwiatkowski et al., 2019), in this setting: (1) the reference corpus does not reliably contain text spans from which the answers can be drawn, and (2) it does not provide…”