Sparse Sequence-to-Sequence Models

By Ben Peters, Vlad Niculae, André F. T. Martins. 2019

301 citations recorded by the source corpus.

Sequence-to-sequence models are a powerful workhorse of NLP. Most variants employ a softmax transformation in both their attention mechanism and output layer, leading to dense alignments and strictly positive output probabilities. This density is wasteful, making models less interpretable and assigning probability mass to many implausible outputs. In this paper, we propose sparse sequence-to-sequence models, rooted in a new family of \alpha-entmax transformations, which includes softmax and sparsemax as particular cases, and is sparse for any \alpha > 1. We provide fast algorithms to evaluate these transformations and their gradients, which scale well for large vocabulary sizes. Our models are able to produce sparse alignments and to assign nonzero probability to a short list of plausible outputs, sometimes rendering beam search exact. Experiments on morphological inflection and machine translation reveal consistent gains over dense models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “, the sparsemax [24] and its generalizations [27, 6, 3].”
Adaptively Sparse Transformers (2019, methodology): “We focus on a recently-introduced ﬂexible family of transformations, α -entmax (Blondel et al., 2019; Peters et al., 2019), deﬁned as: where d := { p ∈ R d : i p i = 1 } is the probability simplex , and, for α ≥ 1 , H T α is the Tsallis continuous family of entropies (Tsallis, 1988): (5) This…”
Surface Form Competition: Why the Highest Probability Answer Isn’t Always Right (2021, background): “These can be reversed by flipping the order of CAUSE and EFFECT, then substituting Multiple Choice Accuracy on GPT-2 Percent of Ties or Wins by Method the natural inverse relation (“because”−→“so” and “so”−→“because” ).”
The SIGMORPHON 2020 Shared Task on Multilingual Grapheme-to-Phoneme Conversion (2020, methodology): “Their submissions either use LSTM-or transformer-based encoder-decoder sequence-to-sequence models with different values of a hyperparameter enforcing sparsity in the ﬁnal layer (Peters et al. 2019).”
SMYRF: Efficient Attention using Asymmetric Clustering (2020, background): “The fact that attention maps of pre-trained layers are sparse is well-known [15, 16, 3, 17, 56, 57].”
$O(n)$ Connections are Expressive Enough: Universal Approximability of Sparse Transformers (2020, background): “This line of works modifies σS in (1a) to other probability maps, by using top-k elements or adopting sparser variants such as sparselin-gen or α-entmax [15, 21].”
Word Alignment by Fine-tuning Embeddings on Parallel Corpora (2021, methodology): “In this paper, we propose to use softmax and a sparse variant α-entmax (Peters et al., 2019) to do the normalization.”
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection (2020, methodology): “The team experimented with two modifications of the softmax function: sparsemax (Martins and Astudillo, 2016, deepspin-02-1 ) and 1.5-entmax (Peters et al., 2019, deepspin-01-1 ).”
One-Size-Fits-All Multilingual Models (2020, methodology): “In practice, this happens frequently for morphological inﬂection (Peters et al., 2019), sometimes making beam search exact.”
An Attentive Survey of Attention Models (2019, background): “Distribution functions such as sparsemax [Martins and Astudillo 2016] and sparse entmax [Martins et al. 2020; Peters et al. 2019] are able to produce sparse alignments and assign nonzero probability to only a short list of plausible outputs.”
Dialogue Transformers (2019, unknown): “5 -entmax [21] in the loss function.”
Neural Oblivious Decision Ensembles for Deep Learning on Tabular Data (2019, methodology): “In our work, we advocate the usage of the recent entmax transformation (Peters et al., 2019) to ”soften” decision trees.”
Deep Neural Networks and Tabular Data: A Survey (2021, methodology): “Restrictions apply. end-to-end optimization, NODE utilizes the entmax transformation [97] and soft splits.”
Language Model Evaluation Beyond Perplexity (2021, background): “…a helpful alteration to a standard distribution learned via MLE, which may in turn provide motivation for recent efforts to employ truncated or sparse probability distributions directly at training time, e.g., truncated loss (Kang and Hashimoto, 2020) or α - entmax loss (Peters et al., 2019).”