Can Generative Pre-trained Language Models Serve As Knowledge Bases for Closed-book QA?

By Cunxiang Wang, Pai Liu, Yue Zhang. 2021

95 citations recorded by the source corpus.

Recent work has investigated the interesting question using pre-trained language models (PLMs) as knowledge bases for answering open questions. However, existing work is limited in using small benchmarks with high test-train overlaps. We construct a new dataset of closed-book QA using SQuAD, and investigate the performance of BART. Experiments show that it is challenging for BART to remember training facts in high precision, and also challenging to answer closed-book questions even if relevant knowledge is retained. Some promising directions are found, including decoupling the knowledge memorizing process and the QA finetune process, forcing the model to recall relevant knowledge when question answering.

Subfields: Artificial Intelligence, Molecular Biology

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Molecular Biology

Citations in the corpus

When large language models meet personalization: perspectives of challenges and opportunities (2023, unknown): “The ability of Large Language Models to retrieve factual knowledge as explicit knowledge bases [38], [39], [40], [41], [42], [43], [40], [41], [44], [45], [46] has been stirred discussed, which presents an opportunity to construct more comprehensive knowledge graphs within recommender systems.”
Shepherd Pre-trained Language Models to Develop a Train of Thought: An Iterative Prompting Approach (2022, background): “C L ] 1 6 M ar 2 02 2 nomenon observed in (Wang et al., 2021) which is more generally related to the catastrophic forgetting problem of deep learning models (McCloskey and Cohen, 1989; Kirkpatrick et al., 2017).”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, background): “Such parametric memory could help solve downstream tasks (Wang et al., 2021; Yu et al., 2023; Sun et al., 2023).”
Recent Advances in Natural Language Processing via Large Pre-trained Language Models: A Survey (2021, methodology): “[188] Closed-book QA What is Southern California often abbreviated as? Southern California, often abbreviated SoCal, is .”
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, methodology): “The Alpaca project [13] has been a pioneering effort in instruction tuning of LLaMA [14], setting a precedent for instruction tuning LLMs, followed by Vicunna [15].”
Iteratively Prompt Pre-trained Language Models for Chain of Thought (2022, background): “There is also the concern that ﬁne-tuning PLMs may interfere with their implicit knowledge storage, a phenomenon observed in (Wang et al., 2021) which is more generally related to the catastrophic forgetting problem of deep learning models (McCloskey and Cohen, 1989; Kirkpatrick et al., 2017;…”
Retrieval-Free Knowledge-Grounded Dialogue Response Generation with Adapters (2021, background): “The explorations on closed-book QA task (Petroni et al. 2019; Roberts, Raffel, and Shazeer 2020; Wang, Liu, and Zhang 2021) with large pre-train LMs also indicate the potential of leveraging the knowledge embedded inside the LMs.”
A Unified Encoder-Decoder Framework with Entity Memory (2022, background): “However, studies showed that closed-book models just recalled similar inputs and answers in their pre-training corpus (Wang et al., 2021), and their performances were behind open-book models.”
Relational World Knowledge Representation in Contextual Language Models: A Review (2021, methodology): “One advantage of this approach, similar to methods for retrieving external entity embed-dings (§ 4.3), is that it supports fusion at both the late (Wang et al., 2021d; Daza et al., 2021) and middle (Liu et al., 2021c) stages.”
Exploring Generalization Ability of Pretrained Language Models on Arithmetic and Logical Reasoning (2021, background): “Following (Lewis et al., 2020b) and (Wang et al., 2021), if one test question appears in the questions of train set, we call it as question overlap , otherwise it is question non-overlap .”