Adaptive Input Representations for Neural Language Modeling

By Alexei Baevski, Michael Auli. 2018

440 citations recorded by the source corpus.

We introduce adaptive input representations for neural language modeling which extend the adaptive softmax of Grave et al. (2017) to input representations of variable capacity. There are several choices on how to factorize the input and output layers, and whether to model words, characters or sub-word units. We perform a systematic comparison of popular choices for a self-attentional architecture. Our experiments show that models equipped with adaptive embeddings are more than twice as fast to train than the popular character input CNN while having a lower number of parameters. On the WikiText-103 benchmark we achieve 18.7 perplexity, an improvement of 10.5 perplexity compared to the previously best published result and on the Billion Word benchmark, we achieve 23.02 perplexity.

Subfields: Artificial Intelligence, Large Language Models, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Large Language Models

In Natural Language Processing

Citations in the corpus

XLNet: Generalized Autoregressive Pretraining for Language Understanding (2019, background): “With a deep root in density estimation 4 [4, 32, 24], language modeling has been a rapidly-developing research area [9, 1, 3].”
An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale (2020, neutral): “Layernorm (LN) is applied before every block, and residual connections after every block (Wang et al., 2019; Baevski & Auli, 2019).”
ALBERT: A Lite BERT for Self-supervised Learning of Language Representations (2019, methodology): “We choose to use the same E for all word pieces because they are much more evenly distributed across documents compared to whole-word embedding, where having different embedding size (Grave et al. (2017); Baevski & Auli (2018); Dai et al. (2019) ) for different words is important.”
Transformer-XL: Attentive Language Models beyond a Fixed-Length Context (2019, methodology): “We adopted adaptive softmax and input representations (Baevski and Auli, 2018; Grave et al., 2016a).”
Cloze-driven Pretraining of Self-attention Networks (2019, methodology): “For hyper-parameter and optimization choices we mostly follow Baevski and Auli (2018).”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, background): “, 2019), language modeling (Dauphin et al., 2017; Baevski and Auli, 2019), abstractive document summarization (Fan et al.”
Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “We use prenormalization transformer blocks (Baevski and Auli, 2019; Child et al., 2019) and GELU activations (Hendrycks and Gimpel, 2016).”
Improving language models by retrieving from trillions of tokens (2021, background): “This transformer has 24 layers, 1024 hidden units, 16 heads and a key size of 64, as in Baevski and Auli (2019).”
Generalization through Memorization: Nearest Neighbor Language Models (2019, methodology): “Since the k NN-LM makes no changes to the underlying LM, we take the exact architecture and optimization described by Baevski & Auli (2019) tokens for the rest of the corpora.”
Scaling Autoregressive Video Models (2019, methodology): “Similar to Baevski and Auli [2], we found that applying layer normalization before each block, rather than after each block as proposed by Vaswani et al.”
End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, positive): “We use the GCNN-14B in [25] as our ConvLM, while the Transformer LM is the same as the one trained on Google Billion Words in [28].”
Neural Text Generation with Unlikelihood Training (2019, methodology): “Training As in Baevski and Auli (2019); Radford et al. (2019), we train on ﬁxed-length contiguous sequences, in our case of length 1,536, which was selected based on GPU memory constraints.”
Multimodal Transformer for Unaligned Multimodal Language Sequences (2019, methodology): “In addition to NMT, transformer networks have also been successfully applied to other tasks, including language modeling (Dai et al., 2018; Baevski and Auli, 2019), semantic role labeling (Strubell et al., 2018), word sense disambiguation (Tang et al., 2018), learning sentence representations…”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “We consider two types of language models (LM): a 4-gram model and a Transformer [3] trained on the Librispeech LM corpus.”
Unsupervised Speech Recognition (2021, methodology): “For Librispeech experiments, we also decode with a Transformer language model (Baevski and Auli, 2018) trained on the Librispeech LM corpus using the beam search decoder of Pratap et al.”