Pointer Sentinel Mixture Models

By Stephen Merity, Caiming Xiong, James Bradbury, Richard Socher. 2016

4,314 citations recorded by the source corpus.

Recent neural network sequence models with softmax classifiers have achieved their best language modeling performance only with very large hidden states and large vocabularies. Even then they struggle to predict rare or unseen words even if the context makes the prediction unambiguous. We introduce the pointer sentinel mixture architecture for neural sequence models which has the ability to either reproduce a word from the recent context or produce a word from a standard softmax classifier. Our pointer sentinel-LSTM model achieves state of the art language modeling performance on the Penn Treebank (70.9 perplexity) while using far fewer parameters than a standard softmax LSTM. In order to evaluate how well language models can exploit longer contexts and deal with more realistic vocabularies and larger corpora we also introduce the freely available WikiText corpus.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

A Deep Reinforced Model for Abstractive Summarization (2017, methodology): “Gulcehre et al. (2016) and Merity et al. (2016) combine this pointer mechanism with the original word generation layer in the decoder to allow the model to use either method at each decoding step.”
Universal Language Model Fine-tuning for Text Classification (2018, methodology): “We use the AWD-LSTM language model (Merity et al., 2017a) with an embedding size of 400 , 3 layers, 1150 hidden activations per layer, and a BPTT batch size of 70 .”
Training Compute-Optimal Large Language Models (2022, positive): “On Wikitext103 (Merity et al., 2017), Chinchilla achieves a perplexity of 7.16 compared to 7.75 for Gopher .”
Toolformer: Language Models Can Teach Themselves to Use Tools (2023, methodology): “Finetuning on CCNet leads to slightly improved performance on the CCNet evaluation subset (perplexity improves from 10.6 to 10.5), but slightly deteriorates performance on WikiText (9.9 to 10.3), presumably because the original pretraining data for GPT-J is more similar to WikiText than our subset o”
Reducing Sentiment Bias in Language Models via Counterfactual Evaluation (2019, methodology): “The WikiText-103 dataset (Merity et al., 2016) consists of 28,591 articles and over 100 million tokens extracted from high quality Wikipedia articles.”
A Constructive Prediction of the Generalization Error Across Scales (2019, positive): “We use Transformer-XL for WikiText-103.”
Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism (2019, methodology): “To avoid training set leakage into our downstream tasks we remove the Wikipedia articles present in the WikiText103 test set (Merity et al., 2016).”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “Our best model in this settings use attention lengths of 1,600 and 3,800 on WikiText-103 and enwiki8 respectively.”
The Natural Language Decathlon: Multitask Learning as Question Answering (2018, background): “…set of baselines for decaNLP that combine the basics of sequence-to-sequence learning [Sutskever et al., 2014, Bahdanau et al., 2014, Luong et al., 2015b] with pointer networks [Vinyals et al., 2015, Merity et al., 2017, Gülçehre et al., 2016, Gu et al., 2016, Nallapati et al., 2016], ad- Preprint.”
Can You Tell Me How to Get Past Sesame Street? Sentence-Level Pretraining Beyond Language Modeling (2018, methodology): “DisSent To extract discourse model examples from the WikiText-103 corpus (Merity et al., 2017), we follow the procedure described in Nie et al. (2019) by extracting clause-pairs that follow speciﬁc dependency relationships within the corpus (see Figure 4 in Nie et al., 2019).”
Get To The Point: Summarization with Pointer-Generator Networks (2017, methodology): “The pointer network has been used to create hybrid approaches for NMT (Gul-cehre et al., 2016), language modeling (Merity et al., 2016), and summarization (Gu et al., 2016; Gulcehre et al., 2016; Miao and Blunsom, 2016; Nallapati et al., 2016; Zeng et al., 2016).”
Fine-tuned Language Models for Text Classification (2018, methodology): “In practice, we use variable length backpropagation sequences (Merity et al., 2017a).”
Adaptive Input Representations for Neural Language Modeling (2018, methodology): “The training data of WIKITEXT -103 comprises about 100M tokens and a vocabulary of around 260K, corresponding to types with more than 3 occurrences in the training data (Merity et al., 2016).”
Breaking the Softmax Bottleneck: A High-Rank RNN Language Model (2017, methodology): “Compared to the vanilla AWD-LSTM, though being more expressive, MoC performs only better on PTB, but worse on WT2.”
Unified Scaling Laws for Routed Language Models (2022, methodology): “The variation across tasks are not the same for a and b. e.g. WikiText-103 has higher values for b and lower for a when compared to the validation set.”
Improving language models by retrieving from trillions of tokens (2021, methodology): “, 2020), Wikitext103 (Merity et al., 2017), Curation Corpus (Curation, 2020), Lambada (Paperno et al.”
Challenges in Detoxifying Language Models (2021, methodology): “For evaluation, besides the C4 validation set, we measure the language model performance on the WikiText-103 10https://www.kaggle.com/c/ jigsaw-toxic-comment-classification-challenge dataset (Merity et al., 2016), which contains 60 articles for validation and 60 articles for testing.”
Compressive Transformers for Long-Range Sequence Modelling (2019, background): “Merity et al. (2016) proposed the WikiText-103 dataset, which contains text from a high quality subset of English-language wikipedia articles.”
Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “The following sections report our results on Wikitext-103 (Merity et al., 2016), enwik-8 (Ma-honey, 2011), as well as ImageNet-64 .”
Program Induction by Rationale Generation: Learning to Solve and Explain Algebraic Word Problems (2017, methodology): “As the baseline we use the attention-based sequence to sequence model proposed by Bahdanau et al. (2014), and proposed augmentations, allowing it to copy from the input (Ling et al., 2016) and from the output (Merity et al., 2016).”