Diachronic degradation of language models: Insights from social media

By Kokil Jaidka, Niyati Chhaya, Lyle Ungar. 2018

46 citations recorded by the source corpus.

Natural languages change over time because they evolve to the needs of their users and the socio-technological environment. This study investigates the diachronic accuracy of pre-trained language models for downstream tasks in machine learning and user profiling. It asks the question: given that the social media platform and its users remain the same, how is language changing over time? How can these differences be used to track the changes in the affect around a particular topic? To our knowledge, this is the first study to show that it is possible to measure diachronic semantic drifts within social media and within the span of a few years.

Subfields: Artificial Intelligence, Cultural Studies, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Noah A. Smith
2. Daniel Khashabi
3. Suchin Gururangan
4. Yejin Choi
5. Paul Röttger
6. Janet B. Pierrehumbert
7. Viviana Patti
8. Valerio Basile

In Cultural Studies

1. Noah A. Smith
2. Janet B. Pierrehumbert
3. Paul Röttger
4. Suchin Gururangan

In Large Language Models

1. Noah A. Smith
2. Suchin Gururangan
3. Daniel Khashabi
4. Pierpaolo Basile
5. Viviana Patti
6. Janet B. Pierrehumbert
7. Komal Florio
8. Paul Röttger

In Natural Language Processing

1. Noah A. Smith
2. Daniel Khashabi
3. Suchin Gururangan
4. Viviana Patti
5. Paul Röttger
6. Janet B. Pierrehumbert
7. Valerio Basile
8. Yejin Choi

Citations in the corpus

• Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, background): “In today’s pretraining-finetuning paradigm, this misalignment can affect a pretrained language model—a situation that has received recent attention (Jaidka et al., 2018; Lazaridou et al., 2021; Peters et al., 2018; Raffel et al., 2020; Röttger and Pierrehumbert, 2021)—or the finetuned task model, or…”
• Time of Your Hate: The Challenge of Time in Hate Speech Detection on Social Media (2020, background): “The closest works found in recent literature are [30], where the authors explore the diachronic aspect in the context of user profiling, and [31], who provides a broader view on diachronicity in word embeddings and corpora.”
• Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (2021, methodology): “Similarly, gender and age prediction models trained on one year’s data perform increasingly worse on later years (Jaidka et al., 2018).”