Natural Questions: A Benchmark for Question Answering Research

By Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Michael Collins, Ankur P. Parikh, Chris Alberti, D. Epstein, Illia Polosukhin, Jacob Devlin and 9 more. 2019

4,925 citations recorded by the source corpus.

We present the Natural Questions corpus, a question answering data set. Questions consist of real anonymized, aggregated queries issued to the Google search engine. An annotator is presented with a question along with a Wikipedia page from the top 5 search results, and annotates a long answer (typically a paragraph) and a short answer (one or more entities) if present on the page, or marks null if no long/short answer is present. The public release consists of 307,373 training examples with single annotations; 7,830 examples with 5-way annotations for development data; and a further 7,842 examples with 5-way annotated sequestered as test data. We present experiments validating quality of the data. We also describe analysis of 25-way annotations on 302 examples, giving insights into human variability on the annotation task. We introduce robust metrics for the purposes of evaluating question answering systems; demonstrate high human upper bounds on these metrics; and establish baseline results using competitive methods drawn from related literature.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Information Retrieval & Search, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Mohit Bansal
  2. Luke S. Zettlemoyer
  3. Jianfeng Gao
  4. Percy Liang
  5. Graham Neubig
  6. Yejin Choi
  7. Quoc V. Le
  8. Noah A. Smith

In Computer Vision and Pattern Recognition

  1. Luc Van Gool
  2. Dacheng Tao
  3. Trevor Darrell
  4. Xiaogang Wang
  5. Ming-Hsuan Yang
  6. Jitendra Malik
  7. Andrew Zisserman
  8. Yi Yang

In Information Retrieval & Search

  1. Jimmy J. Lin
  2. Jamie Callan
  3. Luyu Gao
  4. Hannaneh Hajishirzi
  5. Wen-tau Yih
  6. Sebastian Riedel
  7. Gautier Izacard
  8. Danqi Chen

In Evaluation & Benchmarks

  1. Mohit Bansal
  2. Yejin Choi
  3. Hannaneh Hajishirzi
  4. Luke S. Zettlemoyer
  5. Douwe Kiela
  6. Ashish Sabharwal
  7. Samuel R. Bowman
  8. Matt Gardner

Citations in the corpus