The EOS Decision and Length Extrapolation

By Benjamin Newman, John Hewitt, Percy Liang, Christopher D. Manning. 2020

57 citations recorded by the source corpus.

Extrapolation to unseen sequence lengths is a challenge for neural generative models of language. In this work, we characterize the effect on length extrapolation of a modeling decision often overlooked: predicting the end of the generative process through the use of a special end-of-sequence (EOS) vocabulary item. We study an oracle setting - forcing models to generate to the correct sequence length at test time - to compare the length-extrapolative behavior of networks trained to predict EOS (+EOS) with networks not trained to (-EOS). We find that -EOS substantially outperforms +EOS, for example extrapolating well to lengths 10 times longer than those seen at training time in a bracket closing task, as well as achieving a 40% improvement over +EOS in the difficult SCAN dataset length generalization task. By comparing the hidden states and dynamics of -EOS and +EOS models, we observe that +EOS models fail to generalize because they (1) unnecessarily stratify their hidden states by their linear position is a sequence (structures we call length manifolds) or (2) get stuck in clusters (which we refer to as length attractors) once the EOS token is the highest-probability prediction.

Subfields: Artificial Intelligence, Cultural Studies, Large Language Models

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Cultural Studies

In Large Language Models

Citations in the corpus

Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation (2021, methodology): “…previously and concurrently with transformers on other tasks, such as machine translation (Rosendahl et al., 2019; Neishi & Yoshinaga, 2019; Newman et al., 2020; Kiyono et al., 2021), sequence-to-sequence models trained on an artificial dataset (Hupkes et al., 2020), pretrained…”
Investigating the Limitations of Transformers with Simple Arithmetic Tasks (2021, background): “Supported by recent work, we suspect that the problem is related to the bias of selecting the termination (i.e., end-of-sequence) token when the generated sequence becomes longer than those seen during training (Newman et al., 2020).”
Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing (2022, background): “Model predictions are on average shorter than the gold outputs in the test set, suggesting all models have difﬁculty generating sequences longer than those seen during training (Newman et al., 2020).”
Faith and Fate: Limits of Transformers on Compositionality (2023, background): “Collapsed Compositionality and Robustness Implications Transformers today demonstrate undeniably powerful empirical results.”
Investigating the Limitations of the Transformers with Simple Arithmetic Tasks (2021, unknown): “Supported by recent work, we suspect that the problem is related to the bias of selecting the termination (i.e., end-of-sequence) token when the generated sequence becomes longer than those seen during training (Newman et al., 2020).”
Unlocking Compositional Generalization in Pre-trained Models Using Intermediate Representations (2021, background): “On the length split, RIR yields a boost of 40 accuracy points even though generalizing to longer programs is a known challenge for seq2seq models (Newman et al., 2020).”
SHAPE: Shifted Absolute Position Embedding for Transformers (2021, background): “(ii) E XTRAPOLATE : Shift-invariant models are typically evaluated in terms of extrapolation ability (Wang et al., 2021; Newman et al., 2020).”