Mind the Gap: Assessing Temporal Generalization in Neural Language Models

By Angeliki Lazaridou, A. Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Giménez, Cyprien de Masson d'Autume, Tomás Kociský, Sebastian Ruder and 4 more. 2021

298 citations recorded by the source corpus.

Our world is open-ended, non-stationary, and constantly evolving; thus what we talk about and how we talk about it change over time. This inherent dynamic nature of language contrasts with the current static language modelling paradigm, which trains and evaluates models on utterances from overlapping time periods. Despite impressive recent progress, we demonstrate that Transformer-XL language models perform worse in the realistic setup of predicting future utterances from beyond their training period, and that model performance becomes increasingly worse with time. We find that, while increasing model size alone -- a key driver behind recent progress -- does not solve this problem, having models that continually update their knowledge with new information can indeed mitigate this performance degradation over time. Hence, given the compilation of ever-larger language modelling datasets, combined with the growing list of language-model-based NLP applications that require up-to-date factual knowledge about the world, we argue that now is the right time to rethink the static way in which we currently train and evaluate our language models, and develop adaptive language models that can remain up-to-date with respect to our ever-changing and non-stationary world. We publicly release our dynamic, streaming language modelling benchmarks for WMT and arXiv to facilitate language model evaluation that takes temporal dynamics into account.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, methodology): “Lazaridou et al. (2021) show that language models gradually degrade in performance under such a time-stratiﬁed setting, and propose dynamic evaluation (Krause et al., 2018) as a potential mitigation.”
Mass-Editing Memory in a Transformer (2022, background): “, 2020), even very large models are known to lack more specialized knowledge, and they may recall obsolete information if not re-trained periodically (Lazaridou et al., 2021; Agarwal & Nenkova, 2022; Liska et al., 2022).”
Do Language Models Have Beliefs? Methods for Detecting, Updating, and Visualizing Model Beliefs (2021, background): “Notably, there is good evidence that scaling models to larger sizes will not ﬁx these particular problems or may even exacerbate them in cases like imitative falsehoods in QA, so we will likely need an alternative solution (Lazaridou et al., 2021; Gehman et al., 2020; Lin et al., 2021).”
Taxonomy of Risks posed by Language Models (2022, background): “In addition to increasing representation ofmarginalised groups and addressing specific needs in downstream applications [159], approaches to expanding or updating LMs in real-time also include modes that continue to learn online, as changes occur to a training corpus [8, 43, 106, 112, 179].”
A Review on Language Models as Knowledge Bases (2022, background): “There are increasing evidence that show that scaling LMs to larger sizes is not the solution to generating factually correct information (Lazaridou et al., 2021; Gehman et al., 2020; Lin et al., 2021a).”
Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models (2022, unknown): “…such as vocabulary differences (Blitzer et al., 2006), sociolinguistic (Biber, 1988) or demographic (Rickford, 1985; Blodgett et al., 2016) variables, community membership (Lucy and Bamman, 2021), end-tasks (Gururangan et al., 2020), or temporal shifts (Lazaridou et al., 2021; Luu et al., 2021).”
RealTime QA: What's the Answer Right Now? (2022, background): “A ﬂurry of recent work analyzed performance degradation from temporal misalignement between (pre)training and evaluation/deployment on many NLP tasks (Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021; Luu et al., 2022; Onoe et al., 2022) and proposed mitigation methods for temporal…”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, methodology): “While most of the emphasis on temporal misalignment is on how to adapt our stale models/data to the present time (Dhin-gra et al., 2021; Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021), our experiments also show that models trained on newer data can be mis-aligned from the past, as well.”
Retrieval-based Language Models and Applications (2023, background): “Moreover, these parametric LMs are fundamentally incapable of adapting over time (De Cao et al., 2021; Lazaridou et al., 2021; Kasai et al., 2022), often hallucinate (Shus-ter et al., 2021), and may leak private data from the training corpus (Carlini et al., 2021).”
Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptors (2022, unknown): “Many such errors will arise sequentially in deployment, some of which must be addressed quickly without waiting until new training data is collected [16, 21].”
Knowledge Conflicts for LLMs: A Survey (2024, unknown): “It naturally arises in models trained on data collected in the past, as they may not accurately reflect contemporary re-alities (Luu et al., 2021; Lazaridou et al., 2021; Liska et al., 2022; Su et al., 2022).”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, background): “Lazaridou et al. (2021) show that the corpus-level perplexity on documents from beyond LMs’ training period becomes increasingly poor over time.”
Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts (2023, background): “In addition, the parametric memory may become outdated (Lazaridou et al., 2021; De Cao et al., 2021).”