Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media

By Paul Röttger, Janet B. Pierrehumbert. 2021

85 citations recorded by the source corpus.

Language use differs between domains and even within a domain, language use changes over time. For pre-trained language models like BERT, domain adaptation through continued pre-training has been shown to improve performance on in-domain downstream tasks. In this article, we investigate whether temporal adaptation can bring additional benefits. For this purpose, we introduce a corpus of social media comments sampled over three years. It contains unlabelled data for adaptation and evaluation on an upstream masked language modelling task as well as labelled data for fine-tuning and evaluation on a downstream document classification task. We find that temporality matters for both tasks: temporal adaptation improves upstream and temporal fine-tuning downstream task performance. Time-specific models generally perform better on past than on future test sets, which matches evidence on the bursty usage of topical words. However, adapting BERT to time and domain does not improve performance on the downstream task over only adapting to domain. Token-level analysis shows that temporal adaptation captures event-driven changes in language use in the downstream task, but not those changes that are actually relevant to task performance. Based on our findings, we discuss when temporal adaptation may be more effective.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “Most recently, Röttger and Pierrehumbert (2021) analyzed the interplay between temporal adaptation during pretraining and ﬁnetuning, and concluded that while both stages beneﬁt from adaptation separately, adaptation during pretraining does not help the downstream task.”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “Röttger and Pierrehumbert (2021) further study vocabulary composition approaches for improving adaptation to up-to-date corpora.”
RealTime QA: What's the Answer Right Now? (2022, background): “(Karpukhin et al., 2020); RAG: retrieval-augmented generation (Lewis et al., 2020b).”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, methodology): “4, similar to earlier work (Röttger and Pierrehumbert, 2021), shows that models trained on data from the same time period as the test data perform far better than those from the past.”
Temporal Effects on Pre-trained Models for Language Processing Tasks (2021, background): “However, recent work (Röttger and Pierrehumbert, 2021) has shown that temporal adaptation through continual pre-training even on millions of examples has limited benefit.”
Temporal Preference Optimization for Unsupervised Retrieval (2026, unknown): “Temporal relevance has been explored in language models (Lazaridou et al., 2021; R ¨ ottger & Pierrehumbert, 2021; Rosin et al., 2022; Su et al., 2023; Wang et al., 2023).”