A BERT Baseline for the Natural Questions

By Chris Alberti, Kenton C. T. Lee, Michael Collins. 2019

130 citations recorded by the source corpus.

This technical note describes a new baseline for the Natural Questions. Our model is based on BERT and reduces the gap between the model F1 scores reported in the original dataset paper and the human upper bound by 30% and 50% relative for the long and short answer tasks respectively. This baseline has been submitted to the official NQ leaderboard at this http URL. Code, preprocessed data and pretrained model are available at this https URL.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Natural Questions: A Benchmark for Question Answering Research (2019, background): “We also refer readers to Alberti et al. (2019), which incorportates more recent advances in modeling for question answering.”
TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages (2020, methodology): “…of this dataset for well-studied state-of-the-art models, we present results for a baseline that uses the most recently released multilingual BERT (mBERT) 24 (Devlin et al., 2019) in a setup similar to Alberti et al. (2019), in which all languages are trained jointly in a single model (Table 5).”
Knowledge Guided Text Retrieval and Reading for Open Domain Question Answering (2019, background): “Most work concatenates passages into a single sequence (Swayamdipta et al., 2018; Yang et al., 2018; Song et al., 2018) or reads each in parallel (Clark and Gardner, 2018; Alberti et al., 2019; Min et al., 2019b; Wang et al., 2019).”
Multi-passage BERT: A Globally Normalized BERT Model for Open-domain Question Answering (2019, background): “Nogueira et al., 2018; Alberti et al., 2019).”
Pretrained Transformers for Text Ranking: BERT and Beyond (2020, background): “, [Alberti et al., 2019]. A proper treatment of the literature here would take up another volume,119 but see Chen and Yih [2020] for a tutorial on recent developments.”
Better Rewards Yield Better Summaries: Learning to Summarise Without References (2019, methodology): “In line with Alberti et al. (2019), we therefore use a sliding window approach with the off-set size of 128 tokens to encode overlength summaries and news articles.”
ActBERT: Learning Global-Local Video-Text Representations (2020, background): “Recently, there has been rapid progress in self-supervised representation learning for texts [7, 47], where the Bidirectional Encoder Representations from Transformers (BERT) model [7] generalizes remarkably to many natural language tasks, e.g ., question answering [2].”
Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Fly (2019, background): “PLMs are top performers for many tasks, including QA (Kwiatkowski et al., 2019; Alberti et al., 2019).”
ETC: Encoding Long and Structured Inputs in Transformers (2020, methodology): “Inference: Final predictions are then aggregated similarly as in the work of Alberti et al. (2019), but with two improvements: First, instead of predicting start and end of short/long answer separately, we ﬁrst select the best start position, and then select the best end location that occurs after…”
Synthetic QA Corpora Generation with Roundtrip Consistency (2019, methodology): “In the supplementary material we sketch a possible approach, inspired by the method of Balcan and Blum (2005) for learning with labeled and unlabeled data.”
On Identifiability in Transformers (2019, neutral): “A few examples include: BERT for question answering (Alberti et al., 2019) and XL-Net (Yang et al., 2019a) with d s = 512 , or document translation (Junczys-Dowmunt, 2019) with d s = 1000 .”
ETC: Encoding Long and Structured Data in Transformers (2020, background): “Final predictions are then aggregated as in the work of Alberti et al. (2019).”
QED: A Framework and Dataset for Explanations in Question Answering (2020, methodology): “We constructed the data for the study by taking a random set of 50 correct answers, and 50 incorrect guesses from the NQ baseline model (Alberti et al., 2019), on the Natural Questions dev set.”
What Do Models Learn from Question Answering Datasets? (2020, neutral): “Reference Ours SQuAD 76.3 (Liu et al., 2019) 75.6 TriviaQA 56.3 (Yang et al., 2019) 58.7 NQ 52.7 (Alberti et al., 2019) 73.5 QuAC 54.4 (Qu et al., 2019) 33.3 NewsQA 66.8 (Takahashi et al., 2019) 60.1”
An Exploration of Data Augmentation and Sampling Techniques for Domain-Agnostic Question Answering (2019, background): “To our knowledge this technique is still not prevalent in purely extractive question answering, though Alberti et al. (2019) cite it as a key contributor to their strong baseline on Google’s Natural Questions.”
BERT-kNN: Adding a kNN Search Component to Pretrained Language Models for Better QA (2020, background): “PLMs are top performers for many tasks, including QA (Kwiatkowski et al., 2019; Alberti et al., 2019).”
MuMuQA: Multimedia Multi-Hop News Question Answering via Cross-Media Knowledge Extraction and Grounding (2021, unknown): “The multi-hop text-only QA model takes the question, caption and body text as input, and processes each paragraph in the body text independently, similar to Alberti, Lee, and Collins (2019).”
Toward Deconfounding the Effect of Entity Demographics for Question Answering Accuracy (2021, methodology): “For NQ and SQuAD, we use a fine-tuned BERT (Alberti et al., 2019) with curated training data (e.”