Shortformer: Better Language Modeling using Shorter Inputs

By Ofir Press, Noah A. Smith, Michael Lewis. 2021

100 citations recorded by the source corpus.

Increasing the input length has been a driver of progress in language modeling with transformers. We identify conditions where shorter inputs are not harmful, and achieve perplexity and efficiency improvements through two new methods that decrease input length. First, we show that initially training a model on short subsequences before moving on to longer ones both reduces overall training time and, surprisingly, substantially improves perplexity. Second, we show how to improve the efficiency of recurrence methods in transformers, which let models condition on previously processed tokens when generating sequences that exceed the maximal length the transformer can handle at once. Existing methods require computationally expensive relative position embeddings; we introduce a simple alternative of adding absolute position embeddings to queries and keys instead of to word embeddings, which efficiently produces superior results. We show that these recurrent models also benefit from short input lengths. Combining these techniques speeds up training by a factor of 1.65, reduces memory usage, and substantially improves perplexity on WikiText-103, without adding any parameters.

Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Efficiency & Compression

Citations in the corpus

Efficient Large Scale Language Modeling with Mixtures of Experts (2021, methodology): “…between dense and locally banded sparse attention; and (2) we train our models with sinusoidal positional embeddings, following Shortformer (Press et al., 2020).2 We also train MoE models that mirror our dense model configurations (see the third set of columns in Table 1), so that…”
SCROLLS: Standardized CompaRison Over Long Language Sequences (2022, background): “A growing amount of literature shows that predicting the next token is mostly a local task that does not require modeling long-range dependencies (Khandelwal et al., 2018; Sun et al., 2021), and that masking or down-weighting distant tokens can actually improve perplexity (Press et al., 2021a,b).”
EfficientNetV2: Smaller Models and Faster Training (2021, background): “…training: Previous works have proposed different kinds of progressive training, which dynamically change the training settings or networks, for GANs (Karras et al., 2018), transfer learning (Karras et al., 2018), adversarial learning (Yu et al., 2019), and language models (Press et al., 2021).”
General-purpose, long-context autoregressive modeling with Perceiver AR (2022, background): “As discussed in Shortformer (Press et al., 2021), there is a quality vs. speed tradeoff when evaluating long sequences with regard to stride.”
A Token-level Reference-free Hallucination Detection Benchmark for Free-form Text Generation (2021, background): “We leave the first two sentences in the raw text unchanged to serve as the preceding context, so as to avoid the “early token curse” (Press et al., 2020) where tokens are evaluated at the beginning with limited context.”
Lost in the Middle: How Language Models Use Long Contexts (2023, result): “These observations complement prior work, which found that (non-instruction-tuned) language models are biased towards recent tokens (i.e., the end of the input context; Khandelwal et al., 2018; Press et al., 2021).”
NormFormer: Improved Transformer Pretraining with Extra Normalization (2021, result): “Our model architecture differs from Brown et al. (2020) in two ways: (1) we use only dense attention, while they alternate between dense and locally banded sparse attention; (2) we train our models with sinusoidal positional embeddings, following Shortformer (Press et al., 2020b), since early experi”
HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolution (2023, methodology): “Training on shorter sequences initially (followed by longer sequences) was used by [38] to train small scale Transformers and reduce training time, while [28] used sequence length warm-up to address stability on up to 2k tokens.”
Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, methodology): “Validation and test perplexity is measured by predicting the last 256 words out of the input of 512 consecutive words to avoid evaluating tokens in the beginning with limited context (early token curse, Press et al., 2021).”
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers (2023, methodology): “Similarly to sliding window techniques (Press et al., 2020), this approach doubles the cost of inference but improves results.”
Do Long-Range Language Models Actually Use Long-Range Context? (2021, background): “More recently, Press et al. (2020) also observe negligible benefits of long-term context; we step further in this direction by exploring larger models with more fine-grained analysis.”
Better Language Model with Hypernym Class Prediction (2022, background): “Prior work has shown that LM benefits from training on instances of increasing difficulty (Bengio et al., 2009; Press et al., 2021).”