Dynamic Evaluation of Neural Sequence Models

By Ben Krause, Emmanuel Kahembwe, Iain Murray, S. Renals. 2017

149 citations recorded by the source corpus.

We present methodology for using dynamic evaluation to improve neural sequence models. Models are adapted to recent history via a gradient descent based mechanism, causing them to assign higher probabilities to re-occurring sequential patterns. Dynamic evaluation outperforms existing adaptation approaches in our comparisons. Dynamic evaluation improves the state-of-the-art word-level perplexities on the Penn Treebank and WikiText-2 datasets to 51.1 and 44.3 respectively, and the state-of-the-art character-level cross-entropies on the text8 and Hutter Prize datasets to 1.19 bits/char and 1.08 bits/char respectively.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

Character-Level Language Modeling with Deeper Self-Attention (2018, methodology): “Note, our models do not use dynamic evaluation [24] Using the same hyperparameters and training procedure for text8 , we also train and evaluate the T12 and T64 architectures on enwik8 .”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “Moreover, as deep learning models were tuned by extensive hyper-parameter search, increasing the dimension d beyond several hundred is not helpful3 (Merity et al., 2017; Melis et al., 2017; Krause et al., 2017).”
Time-Aware Language Models as Temporal Knowledge Bases (2021, background): “Lazaridou et al. (2021) show that language models gradually degrade in performance under such a time-stratiﬁed setting, and propose dynamic evaluation (Krause et al., 2018) as a potential mitigation.”
Efficient Neural Architecture Search via Parameter Sharing (2018, methodology): “Since the goal of our work is to discover cell architectures, we only employ the standard training and test process on Penn Treebank, and do not utilize post-training techniques such as neural cache (Grave et al., 2017) and dynamic evaluation (Krause et al., 2017).”
DARTS: Differentiable Architecture Search (2018, methodology): “For fair comparison, we do not ﬁnetune our model at the end of the optimization, nor do we use any additional enhancements such as dynamic evaluation (Krause et al., 2017) or continuous cache (Grave et al., 2016).”
Improving Neural Language Modeling via Adversarial Training (2019, methodology): “62 + dynamic evaluation (Krause et al., 2018) AWD-LSTM (Merity et al.”
FRAGE: Frequency-Agnostic Word Representation (2018, methodology): “We test our model and the baselines at several checkpoints used in the baseline papers: without ﬁnetune, with ﬁnetune, with post-process (continuous cache pointer [14] or dynamic evaluation [19]).”
Direct Output Connection for a High-Rank Language Model (2018, methodology): “Krause et al. (2017) proposed dynamic evaluation that updates parameters based on a recent sequence during testing.”
Memory-based Parameter Adaptation (2018, methodology): “(Krause et al., 2017) 59.8 57.7 68.9 66.1 AWD-LSTM + dyn eval (Krause et al., 2017) 51”
Deep Online Learning via Meta-Learning: Continual Adaptation for Model-Based RL (2018, methodology): “This is representative of commonly used dynamic evaluation methods (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017).”
An Analysis of Neural Language Modeling at Multiple Scales (2018, methodology): “Additional improvements in the model can be obtained through dynamic evaluation (Krause et al., 2018) and mixture-of-softmaxes (Yang et al.”
Barack’s Wife Hillary: Using Knowledge Graphs for Fact-Aware Language Modeling (2019, background): “General language modeling Also related are the recent papers proposing modiﬁcations to the AWD-LSTM that improve performance on Wikitext-2 (Gong et al., 2018; Yang et al., 2018; Krause et al., 2018).”
Learning to Adapt in Dynamic, Real-World Environments through Meta-Reinforcement Learning (2018, methodology): “Dynamic evaluation algorithms (Rei, 2015; Krause et al., 2017; 2016; Fortunato et al., 2017), for example, learn an approximate global distribution at training time and adapt those model parameters at test time to fit the current local distribution via gradient descent.”
Dynamic Evaluation of Transformer Language Models (2019, methodology): “Dynamic evaluation and Transformers have each shown their respective capabilities to use thousands of timesteps of context to improve predictions (Krause et al., 2018; Dai et al., 2019), but it is unclear how much overlap there is between the type of long-range dependencies exploited by…”
Continual Adaptation for Efficient Machine Communication (2019, methodology): “Our approach draws upon the idea of dynamically fine-tuning RNNs (Mikolov et al., 2010; Krause et al., 2017), which has successfully explained key patterns of human behavior in selfpaced reading tasks (Van Schijndel and Linzen, 2018).”
Segatron: Segment-Aware Transformer for Language Modeling and Understanding (2020, methodology): “It is worth noting that we do not list methods with additional training data or dynamic evaluation (Krause et al. 2018) which continues training the model on the test set.”