TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models

By Joel Jang, Seonghyeon Ye, Changho Lee, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun Kim, Minjoon Seo. 2022

137 citations recorded by the source corpus.

Language Models (LMs) become outdated as the world changes; they often fail to perform tasks requiring recent factual information which was absent or different during training, a phenomenon called temporal misalignment. This is especially a challenging problem because the research community still lacks a coherent dataset for assessing the adaptability of LMs to frequently-updated knowledge corpus such as Wikipedia. To this end, we introduce TemporalWiki, a lifelong benchmark for ever-evolving LMs that utilizes the difference between consecutive snapshots of English Wikipedia and English Wikidata for training and evaluation, respectively. The benchmark hence allows researchers to periodically track an LM’s ability to retain previous knowledge and acquire updated/new knowledge at each point in time. We also find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost, which verifies that factual knowledge in LMs can be safely updated with minimal training data via continual learning.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

When Not to Trust Language Models: Investigating Effectiveness and Limitations of Parametric and Non-Parametric Memories (2022, background): “Retrieval-augmented LMs. Relying solely on their parameters to encode a wealth of world knowledge requires a prohibitively large number of parameters (Roberts et al., 2020; Brown et al., 2020) and the knowledge can get obsolete quickly (Ka-sai et al., 2022; Jang et al., 2022).”
When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories (2022, background): “Meanwhile, LMs have also been shown to have limited memorization for less frequent entities (Kandpal et al., 2022), are prone to hallucinations (Shuster et al., 2021), and suffer from temporal degradation (Kasai et al., 2022; Jang et al., 2022).”
Entity Cloze By Date: What LMs Know About Unseen Entities (2022, background): “TemporalWiki (Jang et al., 2022) anno-tates new facts/entities based on the differences between Wikidata/English Wikipedia dumps, but does not necessarily reﬂect real-world changes during the time period (e.g., an ancient queen can be added to Wikidata in 2022).”
Knowledge Unlearning for Mitigating Privacy Risks in Language Models (2022, background): “Another line of work simply quantifies how much knowledge is accumulated and forgotten during pretraining by extracting relational knowledge about the world (Petroni et al., 2019; Lazaridou et al., 2021; Jang et al., 2022b,a).”
RealTime QA: What's the Answer Right Now? (2022, background): “…and evaluation/deployment on many NLP tasks (Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021; Luu et al., 2022; Onoe et al., 2022) and proposed mitigation methods for temporal adaptation (Huang and Paul, 2018, 2019; Dhingra et al., 2022; Jang et al., 2022a,b; Lee et al., 2022).”
Continual Learning of Large Language Models: A Comprehensive Survey (2024, unknown): “The articles with only “ ♣ ” [95, 96, 100] means that only existing CL techniques are studied, without proposing new ones, and the articles with only “ ✗ ” [63, 69] means that special aspects of fine-tuning are studied, without using CL techniques.”