Time Waits for No One! Analysis and Challenges of Temporal Misalignment

By Kelvin Luu, Daniel Khashabi, Suchin Gururangan, Karishma Mandyam, Noah A. Smith. 2021

118 citations recorded by the source corpus.

When an NLP model is trained on text data from one time period and tested or deployed on data from another, the resulting temporal misalignment can degrade end-task performance. In this work, we establish a suite of eight diverse tasks across different domains (social media, science papers, news, and reviews) and periods of time (spanning five years or more) to quantify the effects of temporal misalignment. Our study is focused on the ubiquitous setting where a pretrained model is optionally adapted through continued domain-specific pretraining, followed by task-specific finetuning. We establish a suite of tasks across multiple domains to study temporal misalignment in modern NLP systems. We find stronger effects of temporal misalignment on task performance than have been previously reported. We also find that, while temporal adaptation through continued pretraining can help, these gains are small compared to task-specific finetuning on data from the target time period. Our findings motivate continued research to improve temporal robustness of NLP models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, background): “These issues make adapting large LMs less feasible, especially in domains that change frequently over time (Luu et al., 2021).”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, methodology): “We suppose that larger LLMs, due to their enhanced memorization and reasoning capabilities, are more sensitive to incoherent sentences.”
Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Models (2023, unknown): “(2022) argue that hallucination is also present in human-generated corpora (can be reflected as outdated (Liska et al., 2022; Luu et al., 2022), biased (Chang et al.”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, methodology): “In Section 3.1, we ﬁrst describe the process of constructing the training corpora from Wikipedia snapshots.”
When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “, 2021), and suffer from temporal degradation (Luu et al., 2022; Jang et al., 2022).”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “…such as vocabulary differences (Blitzer et al., 2006), sociolinguistic (Biber, 1988) or demographic (Rickford, 1985; Blodgett et al., 2016) variables, community membership (Lucy and Bamman, 2021), end-tasks (Gururangan et al., 2020), or temporal shifts (Lazaridou et al., 2021; Luu et al., 2021).”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “It naturally arises in models trained on data collected in the past, as they may not accurately reflect contemporary re-alities (Luu et al., 2021; Lazaridou et al., 2021; Liska et al., 2022; Su et al., 2022).”
RealTime QA: What's the Answer Right Now? (2022, background): “A flurry of recent work analyzed performance degradation from temporal misalignement between (pre)training and evaluation/deployment on many NLP tasks (Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021; Luu et al., 2022; Onoe et al., 2022) and proposed mitigation methods for temporal adaptatio”
Temporal Preference Optimization for Unsupervised Retrieval (2026, unknown): “Temporal retrieval is important in domains such as news (Litty K Mathews, 2012; Wang et al., 2012; Luu et al., 2022) and law (Schilder & McCulloh, 2005), where the relevance of information depends on its publication date.”