TimeLMs: Diachronic Language Models from Twitter

By Daniel Loureiro, Francesco Barbieri, Leonardo Neves, Luis Espinosa Anke, José Camacho-Collados. 2022

375 citations recorded by the source corpus.

Despite its importance, the time variable has been largely neglected in the NLP and language model literature. In this paper, we present TimeLMs, a set of language models specialized on diachronic Twitter data. We show that a continual learning strategy contributes to enhancing Twitter-based language models’ capacity to deal with future and out-of-distribution tweets, while making them competitive with standardized and more monolithic benchmarks. We also perform a number of qualitative analyses showing how they cope with trends and peaks in activity involving specific named entities or concept drift. TimeLMs is available at github.com/cardiffnlp/timelms.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Sohee Yang
2. Joel Jang
3. Janghoon Han
4. Gyeonghun Kim
5. Seonghyeon Ye
6. Changho Lee
7. Joongbo Shin
8. Minjoon Seo

Citations in the corpus

• TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, methodology): “Moreover, we clarify that not all Wikipedia/Wikidata updates equate to actual updates of world knowledge.”
• ChatGPT: Jack of all trades, master of none (2023, background): “8%) 817 4 339 /118 21 TweetEmoji P EN Emoji prediction No Multiclass classification TweetEval [84] / [85] 2 No 50000 1666 2 0 (0%) 1664 20 10798 /1010 22 TweetSent P EN Sentiment analysis No Multiclass classification TweetEval [84] / [85] 2 No 12283 5143 0 245 (4.”