Location Attention for Extrapolation to Longer Sequences

By Yann Dubois, Gautier Dagan, Dieuwke Hupkes, Elia Bruni. 2019

50 citations recorded by the source corpus.

Neural networks are surprisingly good at interpolating and perform remarkably well when the training set examples resemble those in the test set. However, they are often unable to extrapolate patterns beyond the seen data, even when the abstractions required for such patterns are simple. In this paper, we first review the notion of extrapolation, why it is important and how one could hope to tackle it. We then focus on a specific type of extrapolation which is especially useful for natural language processing: generalization to sequences that are longer than the training ones. We hypothesize that models with a separate content- and location-based attention are more likely to extrapolate than those with common attention mechanisms. We empirically support our claim for recurrent seq2seq models with our proposed attention on variants of the Lookup Table task. This sheds light on some striking failures of neural models for sequences and on possible methods to approaching such issues.

Subfields: Artificial Intelligence, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Model Architectures

Citations in the corpus

\infty-former: Infinite Memory Transformer (2021, methodology): “…language inference; You et al. (2020) proposed the use of hard-coded Gaussian attention as input-agnostic self-attention layer for machine translation; Dubois et al. (2020) proposed using Gaussian attention as a location attention mechanism to improve the model generalization to longer sequences.”
Faith and Fate: Limits of Transformers on Compositionality (2023, background): “[22] Yann Dubois, Gautier Dagan, Dieuwke Hupkes, and Elia Bruni.”
The EOS Decision and Length Extrapolation (2020, methodology): “Evidence from a number of settings shows that processing sequences longer than those seen at training time is a challenge for neural models (Dubois et al., 2020; Ruis et al., 2020; Hupkes et al., 2020; Klinger et al., 2020).”