Dynamic Language Models for Continuously Evolving Content

By Spurthi Amba Hombaiah, Tao Chen, Mingyang Zhang, Michael Bendersky, Marc Najork. 2021

51 citations recorded by the source corpus.

The content on the web is in a constant state of flux. New entities,issues, and ideas continuously emerge, while the semantics of the existing conversation topics gradually shift. In recent years, pre-trained language models like BERT greatly improved the state-of-the-art for a large spectrum of content understanding tasks.Therefore, in this paper, we aim to study how these language models can be adapted to better handle continuously evolving web content.In our study, we first analyze the evolution of 2013 - 2019 Twitter data, and unequivocally confirm that a BERT model trained on past tweets would heavily deteriorate when directly applied to data from later years. Then, we investigate two possible sources of the deterioration: the semantic shift of existing tokens and the sub-optimal or failed understanding of new tokens. To this end, we both explore two different vocabulary composition methods, as well as propose three sampling methods which help in efficient incremental training for BERT-like models. Compared to a new model trained from scratch offline, our incremental training (a) reduces the training costs, (b) achieves better performance on evolving content, and (c)is suitable for online deployment. The superiority of our methods is validated using two downstream tasks. We demonstrate significant improvements when incrementally evolving the model from a particular base year, on the task of Country Hashtag Prediction, as well as on the OffensEval 2019 task.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “While the impact on language modeling itself has been highlighted in recent work (e.g., Lazaridou et al., 2021; Röttger and Pierrehumbert, 2021; Hombaiah et al., 2021), there are several potential problems that are specific to the encoding of factual knowledge:”
Wide Neural Networks Forget Less Catastrophically (2021, background): “The ability of the wider networks to be less forgetful points towards looking at forgetting through the lens of over-parametrization (Kaplan et al., 2020; Lazaridou et al., 2021; Hombaiah et al., 2021).”
TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models (2022, background): “Temporal misalignment can be caused either by (1) the dynamic nature of language (Röttger and Pierrehumbert, 2021; Hombaiah et al., 2021; Rosin et al., 2021; Loureiro et al., 2022) or (2) the update of factual information (Chen et al.”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “Another line of works focuses on temporal domain shift (Hombaiah et al., 2021), which analyzes the effect of pretraining over up-to-date data to the downstream”
StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Models (2022, methodology): “Amba Hombaiah et al. (2021) propose new incremental methods for online BERT training using vocabulary expansion.”