Entity Cloze By Date: What LMs Know About Unseen Entities

By Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, Greg Durrett. 2022

71 citations recorded by the source corpus.

Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. However, in a dynamic world, new entities constantly arise. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity. We evaluate LMs' perplexity on masked spans within these sentences. We show that models more informed about the entities, such as those with access to a textual definition of them, achieve lower perplexity on this benchmark. Our experimental results demonstrate that making inferences about new entities remains difficult for LMs. Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge. Our automatic data collection pipeline can be easily used to continually update our benchmark.

Subfields: Artificial Intelligence, Management Science and Operations Research, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Natural Language Processing

Citations in the corpus

Neural Theory-of-Mind? On the Limits of Social Intelligence in Large LMs (2022, background): “, 2021) or examining how LLMs handle entities and people (Onoe et al., 2022; Schuster and Linzen, 2022), which could shed light onto better data choices and inductive biases towards neural Theory of Mind and social intelligence.”
RealTime QA: What's the Answer Right Now? (2022, background): “A flurry of recent work analyzed performance degradation from temporal misalignement between (pre)training and evaluation/deployment on many NLP tasks (Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021; Luu et al., 2022; Onoe et al., 2022) and proposed mitigation methods for temporal adaptatio”
A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (2023, unknown): “The factual knowledge embedded within LLMs exhibits clear temporal boundaries and can become outdated over time (Onoe et al., 2022; Kasai et al., 2022; Li et al., 2023a).”
Hallucination is Inevitable: An Innate Limitation of Large Language Models (2024, background): “Issues in the data include poor quality [31], misinformation [35], bias [41, 47], and being outdated [33, 43] knowledge.”
Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, background): “This has been shown in many settings, including but not limited to factual reasoning (Kassner & Sch ¨ utze, 2020; Pandia & Ettinger, 2021; Misra et al., 2023), text generation about new entities (Onoe et al., 2022), or even code generation (Jones & Steinhardt, 2022).”