How Can We Know What Language Models Know?

By Zhengbao Jiang, Frank F. Xu, Jun Araki, Graham Neubig. 2019

1,658 citations recorded by the source corpus.

Abstract Recent work has presented intriguing results examining the knowledge contained in language models (LMs) by having the LM fill in the blanks of prompts such as “Obama is a __ by profession”. These prompts are usually manually created, and quite possibly sub-optimal; another prompt such as “Obama worked as a __ ” may result in more accurately predicting the correct profession. Because of this, given an inappropriate prompt, we might fail to retrieve facts that the LM does know, and thus any given prompt only provides a lower bound estimate of the knowledge contained in an LM. In this paper, we attempt to more accurately estimate the knowledge contained in LMs by automatically discovering better prompts to use in this querying process. Specifically, we propose mining-based and paraphrasing-based methods to automatically generate high-quality and diverse prompts, as well as ensemble methods to combine answers from different prompts. Extensive experiments on the LAMA benchmark for extracting relational knowledge from LMs demonstrate that our methods can improve accuracy from 31.1% to 39.6%, providing a tighter lower bound on what LMs know. We have released the code and the resulting LM Prompt And Query Archive (LPAQA) at https://github.com/jzbjyb/LPAQA.

Subfields: Artificial Intelligence, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

Citations in the corpus

Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering (2020, background): “Recently, several works have shown that factual information can be extracted from large scale language models trained on vast quantities of data (Radford et al., 2019; Petroni et al., 2019; Jiang et al., 2019; Talmor et al., 2019).”
Few-shot Learning with Retrieval Augmented Language Models (2022, methodology): “Prompts themselves can be optimized, for example by search (Jiang et al., 2020; Shin et al., 2020) or by only updating parts of the model (Logan et al.”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, background): “…to often be sensitive to the exact wording of their input when deciding whether or not to call an API; this is perhaps unsurprising given that LMs are known to be very sensitive to the prompt they are provided with in both zero-and few-shot settings (Jiang et al., 2020; Schick and Schütze, 2021a).”
How Much Knowledge Can You Pack into the Parameters of a Language Model? (2020, background): “Interestingly, it has also recently been observed that these models can internalize a sort of implicit “knowledge base” after pre-training (Petroni et al., 2019; Jiang et al., 2019; Talmor et al., 2019).”
Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference (2020, background): “Similar to Jiang et al. (2019), future work could look into automatic identification of suitable patterns and verbalizers.”
Augmented Language Models: a Survey (2023, background): “…models reach a certain size (Wei et al., 2022b; Chung et al., 2022), and performance depends heavily on the format in which examples are presented (Jiang et al., 2020; Min et al., 2022), the choice of few-shot examples, and the order in which they are presented (Kumar and Talukdar, 2021; Lu et…”
Large Language Models Are Human-Level Prompt Engineers (2022, background): “…from discrete prompt search methods, such as prompt generation (Gao et al., 2021; Ben-David et al., 2021), prompt scoring (Davison et al., 2019) and prompt paraphrasing (Jiang et al., 2020; Yuan et al., 2021) to optimize instructions by searching directly in the natural language hypothesis space.”
oLMpics-On What Language Model Pre-training Captures (2019, background): “We tried multiple phrasings (Jiang et al., 2019) and chose the one that achieves highest average zeroshot accuracy across all tested LMs.”
The Power of Scale for Parameter-Efficient Prompt Tuning (2021, background): “Finding an optimal prompt thus requires the selection of prompt tokens, through either manual search or non-differentiable search methods (Jiang et al., 2020; Shin et al., 2020).”
Making Pre-trained Language Models Better Few-shot Learners (2021, background): “Several other attempts have been made in addition—yet these approaches either op- erate in limited domains, such as finding patterns to express specific relations (Jiang et al., 2020), or require a large number of examples for gradientguided search (Shin et al., 2020; Zhong et al., 2021).”
It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners (2020, background): “As ﬁnding ways to reformulate tasks as cloze questions that are understood well by LMs is difﬁ-cult (Jiang et al., 2019), Schick and Sch¨utze (2020a) propose P ET , a method that uses knowledge distillation (Hinton et al., 2015) to easily combine several reformulations.”
Reframing Human-AI Collaboration for Generating Free-Text Explanations (2021, background): “In-context learning methods have been shown to have high variance based on hyperparameters including example order, number of examples given, and which examples are given (Jiang et al., 2020; Zhao et al., 2021; Lu et al., 2022).”
Few-Shot Self-Rationalization with Natural Language Prompts (2021, methodology): “When a LM is pretrained with masked language modeling (Devlin et al., 2019) only, an appropriate NL prompt is constructed by adding and inﬁlling masked tokens (Jiang et al., 2020).”
Calibrate Before Use: Improving Few-Shot Performance of Language Models (2021, methodology): “, analyzing their factual (Petroni et al., 2019; Jiang et al., 2020; Shin et al., 2020) or commonsense knowledge (Bosselut et al.”