How Much Knowledge Can You Pack into the Parameters of a Language Model?

By Adam Roberts, Colin Raffel, Noam Shazeer. 2020

1,084 citations recorded by the source corpus.

It has recently been observed that neural language models trained on unstructured text can implicitly store and retrieve knowledge using natural language queries. In this short paper, we measure the practical utility of this approach by fine-tuning pre-trained models to answer questions without access to any external context or knowledge. We show that this approach scales surprisingly well with model size and outperforms models that explicitly look up knowledge on the open-domain variants of Natural Questions and WebQuestions. To facilitate reproducibility and future work, we release our code and trained models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Natural Language Processing

Citations in the corpus

Language Models are Few-Shot Learners (2020, methodology): “We evaluate in the “closed-book” setting (meaning no conditioning information/articles) as suggested by [RRS20].”
Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (2020, methodology): “Févry et al. [14] follow this convention in order to compare with Roberts et al. [46] (See appendix of [14]).”
Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, background): “Building on that observation and the advances in pretraining of natural language processing models, Roberts et al. (2020) introduced a generative model for open domain question answering.”
UnifiedQA: Crossing Format Boundaries With a Single QA System (2020, methodology): “We leave extensions of UNIFIEDQA to other formats such as to direct-answer questions (Roberts et al., 2020) as a promising avenue for future work.”
LaMDA: Language Models for Dialog Applications (2022, positive): “Recent studies on scaling have found that performance on question-answering tasks improves with model size [32, 33], similar to our findings on pre-trained LaMDA prior to fine-tuning.”
Deduplicating Training Data Mitigates Privacy Risks in Language Models (2022, methodology): “Prior work using LMs for closed-book question answering shows that deduplication is not universally beneficial, as memorization of facts from the training data can be necessary for certain tasks (Petroni et al., 2019; Roberts et al., 2020).”
mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer (2020, methodology): “…for the “Text-to-Text Transfer Transformer” (T5) model released by Raffel et al. (2020) have been used to achieve state-of-the-art results on many benchmarks (Khashabi et al., 2020; Roberts et al., 2020; Kale, 2020; Izacard and Grave, 2020; Nogueira et al., 2020; Narang et al., 2020, etc.).”
Entities as Experts: Sparse Memory Access with Entity Supervision (2020, methodology): “Recent work has shown that these models capture some declarative knowledge, and has suggested that neural language models could take the place of curated knowledge bases or textual corpora for tasks such as question answering (Petroni et al., 2019; Roberts et al., 2020).”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, background): “…correctness of text decoded from modern LMs. Additionally, recent works (Petroni et al., 2019; Logan et al., 2019; Broscheit, 2019; Roberts et al., 2020) demonstrate that beyond general linguistic capabilities, language models can also pick up factual knowledge present in the…”
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, methodology): “It is worth noting that stronger open-domain QA performance have been achieved with much larger pretrained model (Raffel et al., 2019), i.e. , T5 (Roberts et al., 2020) or better designed pretraining paradigm combined with more updates, i.e. , REALM (Guu et al., 2020).”
Autoregressive Search Engines: Generating Substrings as Document Identifiers (2022, background): “Some works have demonstrated that knowledge stored in the parameters of these models can be retrieved to some extend by directly generating evidence given a query (Petroni et al., 2019, 2020; Roberts et al., 2020).”
You Only Need One Model for Open-domain Question Answering (2021, methodology): “On the other hand, MSS trains the model by ‘ input-output ’ pairs that have strong supervision for the answer generation, but no supervision for the retriever, requiring additional warm-up training for retrieval such as ICT.”
Learning to Retrieve Passages without Supervision (2021, methodology): “…(Lee et al., 2019; Chang et al., 2020; Guu et al., 2020; Sachan et al., 2021a; Gao and Callan, 2021a), we attempt to bridge the gap between unsupervised dense models and strong sparse (e.g. BM25; Robertson and Zaragoza 2009) or supervised dense baselines (e.g. DPR; Karpukhin et al. 2020).”
R2-D2: A Modular Baseline for Open-Domain Question Answering (2021, background): “The generative reader generates an answer while conditioned on question alone (Roberts et al., 2020), or question with relevant passages (Lewis et al.”
End-to-End Training of Multi-Document Reader and Retriever for Open-Domain Question Answering (2021, methodology): “We compare with the work of Roberts et al. (2020) who show that larger T5 models—when finetuned with question-answer pairs—can perform remarkably well.”