Measuring and Improving Consistency in Pretrained Language Models

By Yanai Elazar, Nora Kassner, Shauli Ravfogel, Abhilasha Ravichander, Eduard H. Hovy, Hinrich Schütze, Yoav Goldberg. 2021

487 citations recorded by the source corpus.

Abstract Consistency of a model—that is, the invariance of its behavior under meaning-preserving alternations in its input—is a highly desirable property in natural language processing. In this paper we study the question: Are Pretrained Language Models (PLMs) consistent with respect to factual knowledge? To this end, we create ParaRel🤘, a high-quality resource of cloze-style query English paraphrases. It contains a total of 328 paraphrases for 38 relations. Using ParaRel🤘, we show that the consistency of all PLMs we experiment with is poor— though with high variance between relations. Our analysis of the representational spaces of PLMs suggests that they have a poor structure and are currently not suitable for representing knowledge robustly. Finally, we propose a method for improving model consistency and experimentally demonstrate its effectiveness.1

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Safety, Robustness & Interpretability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Safety, Robustness & Interpretability

Citations in the corpus

Self-Consistency Improves Chain of Thought Reasoning in Language Models (2022, background): “, 2020), and factual knowledge extraction (Elazar et al., 2021).”
Locating and Editing Factual Associations in GPT (2022, background): “Some research has been done for masked models (Petroni et al., 2019; Jiang et al., 2020; Elazar et al., 2021a; Geva et al., 2021; Dai et al., 2022; De Cao et al., 2021), but GPT has architectural differences such as unidirectional attention and generation capabilities that provide an opportunity…”
Mass-Editing Memory in a Transformer (2022, background): “However, LLM knowledge is only implicit; responses are sensitive to specific phrasings of the prompt (Elazar et al., 2021; Petroni et al., 2020), and it remains difficult to catalog, add, or update knowledge (AlKhamissi et al., 2022).”
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, methodology): “Paraphrase Consistency is the fraction of paraphrase pairs for which a model produces the same output (Elazar et al., 2021).”
Knowledge Neurons in Pretrained Transformers (2021, methodology): “Elazar et al. (2021) measure and improve the consistency of pretrained models with respect to factual knowledge prediction.”
Unsolved Problems in ML Safety (2021, background): “Researchers could create evaluation schemes that catch models being inconsistent [50], as inconsistency implies that they did not assert only what they believe.”
Enriching a Model's Notion of Belief using a Persistent Memory (2021, background): “, 2020), their answers to probing questions can be inconsistent (Elazar et al., 2021; Ravichander et al., 2020; Kassner and Schütze, 2020), making it hard to pin down what a model actually “believes””
Can We Edit Factual Knowledge by In-Context Learning? (2023, background): “However, the wide application of LMs also raises growing concerns regarding its pitfall of generating content that is fake (Elazar et al., 2021; Cao et al., 2021a), out-dated (Dhingra et al., 2022), biased (Sheng et al., 2019; Zhao et al., 2021), and offensive (Gehman et al., 2020).”
Moving the Eiffel Tower to ROME: Tracing and Editing Facts in GPT (2021, methodology): “Each record in C OUNTER F ACT is derived from a corresponding entry in P ARA R EL (Elazar et al., 2021), which comes with a knowledge tuple t = ( s, r, o ) , as well as P ( s, r ) , a collection of hand-curated, semantically-equivalent prompts.”
Locating and Editing Factual Knowledge in GPT (2022, methodology): “Research has been done for masked models (Petroni et al., 2019; Jiang et al., 2020; Elazar et al., 2021a; Geva et al., 2021; Dai et al., 2021; De Cao et al., 2021), but GPT’s architectural differences (e.g., unidirectional attention, generation capabilities) provide an opportunity for new insights.”
Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations (2022, background): “This is in part due to the broader limitations of generative LMs, which assign high probability to unlikely sentences (Welleck et al., 2020; Holtzman et al., 2021) and are sensitive to semantic perturbations (Elazar et al., 2021).”
BeliefBank: Adding Memory to a Pre-Trained Language Model for a Systematic Notion of Belief (2021, background): “…PTLMs contain extensive world knowledge (Petroni et al., 2019; Roberts et al., 2020), they can be inconsistent in their answers to probing questions (Ettinger, 2020; Davison et al., 2019; Ravichander et al., 2020; Elazar et al., 2021; Subramanian et al., 2020), making their “world model” unclear.”
XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models (2023, background): “Many other works have found similar instability in LLM evaluations (e.g. Elazar et al., 2021; Wang et al., 2021, 2023; Röttger et al., 2024; Wang et al., 2024).”
True Few-Shot Learning with Prompts—A Real-World Perspective (2021, background): “, 2020; Webson and Pavlick, 2021) and that they are highly sensitive to the exact wording of the instructions provided (Jiang et al., 2020; Schick and Schütze, 2020; Chu et al., 2021; Elazar et al., 2021).”
Prompt Consistency for Zero-Shot Task Generalization (2022, methodology): “With that in 322 mind, we focus on Fleiss’ kappa (Fleiss, 1971), a 323 commonly used metric to assess the reliability of 324 agreement between a fixed number of raters.”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, methodology): “Prior work employs various heuristics, such as negation injection (Niu & Bansal, 2018; Kassner et al., 2021; Gubelmann & Handschuh, 2022) and entity substitution (Longpre et al., 2021; Zhou et al., 2023), and finds that language models (both large and small) tend to be stubborn and cling to their…”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “Elazar et al. (2021) found that BERT, RoBERTa, and ALBERT struggle with knowledge consistency, achieving accuracy rates barely exceeding 50-60%.”
Large Language Model Alignment: A Survey (2023, background): “Unfaithful Content Yet another problem (Elazar et al., 2021; Ji et al., 2023; Liu et al., 2023d) that hinders the large-scale deployment of LLMs is their tendency to generate unfaithful or even fabricated content, known as misinformation (Branwen, 2020; Dale, 2021; Rae et al.”
A Review on Language Models as Knowledge Bases (2022, methodology): “To that end, several benchmarks have been proposed to measure consistency of LMs (Ravichander et al., 2020; Elazar et al., 2021), while other datasets have been adapted for that purpose (Levy et al., 2017; Wang et al., 2021c).”