On the State of the Art of Evaluation in Neural Language Models

By Gábor Melis, Chris Dyer, Phil Blunsom. 2017

557 citations recorded by the source corpus.

Ongoing innovations in recurrent neural network architectures have provided a steady influx of apparently state-of-the-art results on language modelling benchmarks. However, these have been evaluated using differing code bases and limited computational resources, which represent uncontrolled sources of experimental variation. We reevaluate several popular architectures and regularisation methods with large-scale automatic black-box hyperparameter tuning and arrive at the somewhat surprising conclusion that standard LSTM architectures, when properly regularised, outperform more recent models. We establish a new state of the art on the Penn Treebank and Wikitext-2 corpora, as well as strong baselines on the Hutter Prize dataset.

Subfields: Artificial Intelligence, Natural Language Processing, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Evaluation & Benchmarks

Citations in the corpus

Deep Contextualized Word Representations (2018, background): “…the probability of to-ken t k given the history ( t 1 , ..., t k − 1 ) : Recent state-of-the-art neural language models (J´ozefowicz et al., 2016; Melis et al., 2017; Mer-ity et al., 2017) A backward LM is similar to a forward LM, except it runs over the sequence in reverse, predicting the…”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “Moreover, as deep learning models were tuned by extensive hyper-parameter search, increasing the dimension d beyond several hundred is not helpful3 (Merity et al., 2017; Melis et al., 2017; Krause et al., 2017).”
ByT5: Towards a Token-Free Future with Pre-trained Byte-to-Byte Models (2021, methodology): “This precedent led many to use character-level language modeling as a benchmark to evaluate neural architectures (Kalchbrenner et al., 2016; Chung et al., 2017; Ha et al., 2017; Zilly et al., 2017; Melis et al., 2018; Al-Rfou et al., 2019).”
How Decoding Strategies Affect the Verifiability of Generated Text (2019, methodology): “…w2, . . . , wt−1], autoregressive LMs commonly estimate the probability distribution of the next target using neural models (Mikolov and Zweig, 2012; Melis et al., 2017; Bengio et al., 2003) with: p(wt | ct) = softmax(Wht + b) (1) where ht ∈ Rk is the output vector of a neural network at position…”
Language Models as Knowledge Bases? (2019, background): “…2003; Mikolov and Zweig, 2012), convolutional layers (Dauphin et al., 2017), recurrent neural networks (Zaremba et al., 2014; Merity et al., 2016; Melis et al., 2017) or self-attention mechanisms (Radford et al., 2018; Dai et al., 2019; Radford et al., 2019 BERT : Instead of a standard…”
How Can We Know What Language Models Know? (2019, background): “Recent years have seen the primary role of language models (LMs) transition from generating or evaluating the fluency of natural text (Mikolov and Zweig, 2012; Merity et al., 2018; Melis et al., 2018; Gamon et al., 2005) to being a powerful tool for text understanding.”
Elucidating image-to-set prediction: An analysis of models, losses and datasets (2019, background): “This lack of rigour leads to contributions with apparent new state-of-the-art results, which are evaluated under different dataset splits, evaluation metrics, experimental budgets [43], and other uncontrolled sources of variations [43, 3].”
Measuring Data (2022, methodology): “It was subsequently leveraged to evaluate the performance of language models (Brown et al., 1992; Bengio et al., 2000; Melis et al., 2017) as well as datasets, for tasks such as perplexity sampling (De la Rosa et al., 2022) and data selection (Toral et al., 2015).”
On Network Design Spaces for Visual Recognition (2019, background): “There has been an encouraging recent trend toward better reproducibility in machine learning [25, 22, 9].”
Green AI (2019, methodology): “With a ﬁxed architecture, researchers from DeepMind [25] evaluated 1,500 hyperparameter assignments to demonstrate that an LSTM language model [15] can reach state-of-the-art perplexity results.”
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping (2020, methodology): “Melis et al. (2018) heavily tuned the hyperpamareters of an LSTM language model, for some experiments running 1,500 rounds of Bayesian optimization (thus, training 1,500 models).”
Regularizing and Optimizing LSTM Language Models (2017, methodology): “…to generate an RNN cell tailored to the speciﬁc task of language modeling, with the cell far more complex than the LSTM. Independently of our work, Melis et al. (2017) apply extensive hyperparameter search to an LSTM based language modeling implementation, analyzing the sensitivity of RNN based…”
Show Your Work: Improved Reporting of Experimental Results (2019, methodology): “2186 tained purely through more intensive hyperparameter tuning (Melis et al., 2018; Lipton and Steinhardt, 2018).”
Targeted Syntactic Evaluation of Language Models (2018, background): “While a large number of alternative architectures have been proposed in the past few years, LSTMs are still highly competitive (Melis et al., 2018).”
DARTS: Differentiable Architecture Search (2018, methodology): “Note that our automatically searched cell outperforms the extensively tuned LSTM (Melis et al., 2018), demonstrating the importance of architecture search in addition to hyperparameter search.”
Neural Architecture Optimization (2018, methodology): “Models and Techniques #params Test Perplexity Variational LSTM + weight tying [20] 28M 87.0 LSTM + continuos cache pointer [16] - 68.9 LSTM [32] 33 66.0 4-layer LSTM + skip connection + averaged weight drop + weight penalty + weight tying [31] 24M 65.9”
Neural Optimizer Search with Reinforcement Learning (2017, methodology): “Weight decay, embedding, state and output dropout, state reset probability and embedding ratio are tuned with a black-box hyperparam-eter tuner (Golovin et al., 2017), similarly to Melis et al. (2017).”