Time-Aware Language Models as Temporal Knowledge Bases

By Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, D. Gillick, Jacob Eisenstein, William W. Cohen. 2021

394 citations recorded by the source corpus.

Many facts come with an expiration date, from the name of the President to the basketball team Lebron James plays for. However, most language models (LMs) are trained on snapshots of data collected at a specific moment in time. This can limit their utility, especially in the closed-book setting where the pretraining corpus must contain the facts the model should memorize. We introduce a diagnostic dataset aimed at probing LMs for factual knowledge that changes over time and highlight problems with LMs at either end of the spectrum—those trained on specific slices of temporal data, as well as those trained on a wide range of temporal data. To mitigate these problems, we propose a simple technique for jointly modeling text with its timestamp. This improves memorization of seen facts from the training time period, as well as calibration on predictions about unseen facts from future time periods. We also show that models trained with temporal context can be efficiently “refreshed” as new data arrives, without the need for retraining from scratch.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

Citations in the corpus

Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “Even for bigger models, the gap between predictions with and without API calls remains high. closer inspection shows that improvements on T EMP LAMA can not be attributed to the calendar tool, which is only used for 0.2% of all examples, but mostly to the Wikipedia search and question answering tool”
Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey (2022, background): “Future work may benefit from drawing on continual (Dhingra et al., 2022) and reinforcement”
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, background): “These are important applications given that LMs (1) struggle with future data when trained on data from the past (Lazaridou et al., 2021; Dhingra et al., 2021), (2) generate morally undesirable text in many situations (Gehman et al., 2020; Bender et al., 2021), and (3) simply give inaccurate…”
ELLE: Efficient Lifelong Pre-training for Emerging Data (2022, background): “tual knowledge that changes over time (Dhingra et al., 2021; Jang et al., 2021).”
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (2022, background): “…answering questions about knowledge with temporal grounding of facts in a knowledge graph (Saxena et al., 2021; Jia et al., 2021) or news articles (Wang et al., 2021; Dhingra et al., 2021) but without grounding the questions with a question date; or have repurposed questions from ar X iv :2 20 5.”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “While most of the emphasis on temporal misalignment is on how to adapt our stale models/data to the present time (Dhingra et al., 2021; Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021), our experiments also show that models trained on newer data can be misaligned from the past, as well.”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, background): “Newer cloze benchmarks (Dhingra et al., 2021; Jang et al., 2021) integrate temporal aspects to identify a time period when a cloze sentence is valid, but do not differentiate new and existing entities.”
Mind's Eye: Grounded Language Model Reasoning through Simulation (2022, background): “These methods are inherently limited as their reasoning ability completely relies on the knowledge perpetuated in the LM—their performance could suffer if the knowledge learnt by the LM is incorrect (Petroni et al., 2019) or outdated (Dhingra et al., 2022).”
Can We Edit Factual Knowledge by In-Context Learning? (2023, methodology): “However, the wide application of LMs also raises growing concerns regarding its pitfall of generating content that is fake (Elazar et al., 2021; Cao et al., 2021a), out-dated (Dhingra et al., 2022), biased (Sheng et al., 2019; Zhao et al., 2021), and offensive (Gehman et al., 2020).”
Making Retrieval-Augmented Language Models Robust to Irrelevant Context (2023, background): “…open-domain question answering (ODQA; Chen et al. 2017) and other knowledge-intensive tasks (Thorne et al., 2018; Petroni et al., 2021) require vast amounts of up-to-date factual knowledge about rare entities that even very large models cannot memorize (Roberts et al., 2020; Dhingra et al., 2022).”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, background): “Dhingra et al. (2021)165 have proposed explicitly including time informa-166 tion during pretraining as a potential solution.”
RealTime QA: What's the Answer Right Now? (2022, background): “, 2022) and proposed mitigation methods for temporal adaptation (Huang and Paul, 2018, 2019; Dhingra et al., 2022; Jang et al., 2022a,b; Lee et al., 2022).”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, neutral): “During training, they inject temporal information into training examples as prefixes of prompts, using special tokens [198], explicit year information [52], or syntax-guided structural information [219].”
ToolQA: A Dataset for LLM Question Answering with External Tools (2023, background): “For example, there are works that study how text retrievers augment LLMs’ performance on open-domain question-answering [21, 74], fact-checking [60], and timely information benchmarks [7, 23, 78, 12].”