Pitfalls of Static Language Modelling

By Angeliki Lazaridou, A. Kuncoro, Elena Gribovskaya, Devang Agrawal, Adam Liska, Tayfun Terzi, Mai Giménez, Cyprien de Masson d'Autume, Sebastian Ruder, Dani Yogatama and 4 more. 2021

54 citations recorded by the source corpus.

Subfields: Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Large Language Models

Citations in the corpus

LaMDA: Language Models for Dialog Applications (2022, background): “Lazaridou et al. (2021) call this the temporal generalization problem [97].”
Improving language models by retrieving from trillions of tokens (2021, background): “Those stem from their ability to memorise training data, their high training cost, the static nature of their training data (Lazaridou et al., 2021), their tendency of amplifying inherent biases in the training data, and their ability to generate toxic language (Gehman et al.”
Ethical and social risks of harm from Language Models (2021, background): “Transformer models have been shown to perform worse when applied to utterances from a different period to the time when their training data was generated (Lazaridou et al., 2021).”
Anticipating Safety Issues in E2E Conversational AI: Framework and Tooling (2021, background): “In addition to evolving benchmarks, we might also consider evolving models: most current LLMs are static and thus unable to represent value change (Lazaridou et al., 2021).”
Time-Aware Language Models as Temporal Knowledge Bases (2021, methodology): “Lazaridou et al. (2021) show that language models gradually degrade in performance under such a time-stratiﬁed setting, and propose dynamic evaluation (Krause et al., 2018) as a potential mitigation.”
Internet-Augmented Dialogue Generation (2021, methodology): “…of the world, where new information is coming in by the day – or even by the minute – as the knowledge in static models is gleaned from the point in time when the dataset was collected, and then frozen into the model that is trained; see (Lazaridou et al., 2021) for criticisms of this approach.”
Wide Neural Networks Forget Less Catastrophically (2021, background): “Machine learning is relying more and more on training large models on large static datasets to reach impressive results (Kaplan et al., 2020; Lazaridou et al., 2021; Hom-baiah et al., 2021).”
Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Models (2022, background): “Tem-porality can be important when considering offensive terms or when evaluating truthfulness in areas which are rapidly evolving, e.g., in a pandemic scientiﬁc understanding and medical advice can change quickly [67].”
DEMix Layers: Disentangling Domains for Modular Language Modeling (2021, background): “This design principle emphasizes the ability to rapidly adapt the LM after training, a need that has been broadly advocated for language systems (Dinan et al., 2021; Lazaridou et al., 2021).”
Dynamic Language Models for Continuously Evolving Content (2021, background): “The most relevant work is by [25] who demonstrate that Transformer-XL [10] (a left-to-right autoregressive language model) handles semantic shifts poorly in news and scientific domains, and highlight the importance of adapting language models to continuous stream of new information.”
Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (2021, background): “This is aligned in terms of order of magnitude with previous work on adapting pretrained language models (Gururangan et al., 2020; Lazaridou et al., 2021).”
Better Language Model with Hypernym Class Prediction (2022, methodology): “AR X IV (Lazaridou et al., 2021)”
The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation (2021, methodology): “Validation & testing The need for multiple test sets to capture performance on a variety of down-stream conditions has already been recommended by recent work (Søgaard et al., 2021; Lazaridou et al., 2021).”
Time Waits for No One! Analysis and Challenges of Temporal Misalignment (2021, methodology): “In today’s pretraining-finetuning paradigm, this misalignment can affect a pretrained language model—a situation that has received recent attention (Jaidka et al., 2018; Lazaridou et al., 2021; Peters et al., 2018; Raffel et al., 2020; Röttger and Pierrehumbert, 2021)—or the finetuned task model, or”
TimeLMs: Diachronic Language Models from Twitter (2022, background): “However, currently, both from model development and evaluation standpoints, this paradigm is essentially static, which affects both the ability to generalize to future data and the reliability of experimental results, since it is not uncommon that evaluation benchmarks overlap with pre-training corp”
Randomness In Neural Network Training: Characterizing The Impact of Tooling (2021, methodology): “…depth of model [Snapp and Shamir, 2021, Shamir et al., 2020], hyper-parameter choices [Lucic et al., 2018, Henderson et al., 2017, Kadlec et al., 2017], the use of data parallelism [Shallue et al., 2019] and test set construction [Søgaard et al., 2021, Lazaridou et al., 2021, Melis et al., 2018].”
SituatedQA: Incorporating Extra-Linguistic Contexts into QA (2021, background): “Recent work (Lazaridou et al., 2021) studied temporal drift in the context of large scale pretrained language model, hinting reusing models from previous snap-shot can cause performance decay, as we observe in our study as well.”
DEBAGREEMENT: A comment-reply dataset for (dis)agreement detection in online debates (2021, methodology): “This follows a realistic setting in which one uses a trained model to perform prediction on new, incoming data [24].”
Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora (2021, background): “…knowledge distillation (KD) (Hinton et al., 2015) techniques have been studied intensively for pretrained language models (Sun et al., 2019), applying them to continual learning has been under-explored outside image classiﬁcation tasks (Li and Hoiem, 2018; Rebufﬁ et al., 2017; Hou et al., 2018).”