Dynamic Evaluation of Transformer Language Models

By Ben Krause, Emmanuel Kahembwe, Iain Murray, S. Renals. 2019

48 citations recorded by the source corpus.

This research note combines two methods that have recently improved the state of the art in language modeling: Transformers and dynamic evaluation. Transformers use stacked layers of self-attention that allow them to capture long range dependencies in sequential data. Dynamic evaluation fits models to the recent sequence history, allowing them to assign higher probabilities to re-occurring sequential patterns. By applying dynamic evaluation to Transformer-XL models, we improve the state of the art on enwik8 from 0.99 to 0.94 bits/char, text8 from 1.08 to 1.04 bits/char, and WikiText-103 from 18.3 to 16.4 perplexity points.

Subfields: Artificial Intelligence, Large Language Models, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Evaluation & Benchmarks

Citations in the corpus

Compressive Transformers for Long-Range Sequence Modelling (2019, methodology): “valuation approach of loading a model checkpoint and continuing training over one epoch of the test set, then we obtain a test perplexity of 16.1. This is slightly better than the published 16.4 from Krause et al. (2019) — which uses a more sophisticated dynamic evaluation approach on top of the Tra”
Generalization through Memorization: Nearest Neighbor Language Models (2019, methodology): “We compare with models trained only on the standard training set, but recent work has shown performance can be improved by training on additional data, from either the test set (Krause et al., 2019) or large amounts of web text (Shoeybi et al., 2019).”
Mind the Gap: Assessing Temporal Generalization in Neural Language Models (2021, methodology): “We then explore a simple yet effective way of keeping our models up-to-date by continually updating the model’s parameters through dynamic evaluation (Mikolov et al., 2010; Krause et al., 2019), which performs a few steps of gradient descent on streams of new data (§6), and outline other promising a”
Vision-Language Transformer and Query Generation for Referring Segmentation (2021, background): “The transformer model quickly gain its attraction in the Natural Language Processing (NLP) and shows promising performance on several major NLP tasks such as machine translation [25], language modeling [14], question answering [2].”
Learning to (Learn at Test Time): RNNs with Expressive Hidden States (2024, unknown): “More recently, the same idea has also been applied to natural language processing, where it is called dynamic evaluation [47, 48].”
Learning Distributed Representations of Symbolic Structure Using Binding and Unbinding Operations (2018, unknown): “very well on the dev test, so we omitted the results from GRU in the table. model WikiText-103 WikiText-2 #Params Two-layer Architecture - dim 1024 TPRU 41.4 76.8 12.6m LSTM 42.3 76.0 16.8m SOTA 16.4 [17] 39.1 [10] 257m / 35m (a) 0.0 2.5 5.0 7.5 10.0 12.5 15.0 17.5 Epochs 3.5 4.0 4.5 5.0 5.5 6.0 Neg”
Temporal Adaptation of BERT and Performance on Downstream Document Classification: Insights from Social Media (2021, background): “and demonstrate that dynamic evaluation (Krause et al., 2019), a form of unsupervised online learning, mitigates this degradation.”
Just Add Functions: A Neural-Symbolic Language Model (2019, background): “Neural network language models (NNLMs) have achieved ever-improving accuracy due to more sophisticated architectures and increasing amounts of training data (Radford et al. 2019; Krause et al. 2019; Merity, Keskar, and Socher 2017).”