From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

By André F. T. Martins, Ramón Fernandez Astudillo. 2016

897 citations recorded by the source corpus.

We propose sparsemax, a new activation function similar to the traditional softmax, but able to output sparse probabilities. After deriving its properties, we show how its Jacobian can be efficiently computed, enabling its use in a network trained with backpropagation. Then, we propose a new smooth and convex loss function which is the sparsemax analogue of the logistic loss. We reveal an unexpected connection between this new loss and the Huber classification loss. We obtain promising empirical results in multi-label classification problems and in attention-based neural networks for natural language inference. For the latter, we achieve a similar performance as the traditional softmax, but with a selective, more compact, attention focus.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning (2021, neutral): “, the sparsemax [24] and its generalizations [27, 6, 3].”
Attention is not Explanation (2019, neutral): “, (Xu et al., 2015; Choi et al., 2016; Lei et al., 2017; Martins and Astudillo, 2016; Xie et al., 2017; Mullenbach et al., 2018).”
RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism (2016, methodology): “In Step 2, we can use Sparsemax [28] instead of Softmax for sparser attention weights.”
Is Attention Interpretable? (2019, methodology): “Given attention layer ℓ ’s learned parameters, element 𝑖 of a sequence, and its encoded representation 𝐡 𝑖 , the attention weight 𝛼 𝑖 is computed using ℓ ’ s learned context vector 𝐜 ℓ as follows: 𝑗 4 Alternatives such as sparse attention (Martins and Astudillo, 2016) and unnormalized attenti”
Adaptively Sparse Transformers (2019, methodology): “The Transformer architecture (Vaswani et al., 2017) for deep neural networks has quickly risen to prominence in NLP through its efﬁciency and performance, leading to improvements in the state of the art of Neural Machine Translation (NMT; Junczys-Dowmunt et al., 2018; Ott et al., 2018), as well as i”
Sparse Sinkhorn Attention (2020, methodology): “Moreover, hard attention can also improve inference speeds, as demonstrated by methods such as Sparsemax (Martins & Astudillo, 2016).”
Interpretable Neural Predictions with Differentiable Binary Variables (2019, background): “They derive a number of relaxations to max , including softmax itself, but in particular, they target relaxations such as sparsemax (Martins and Astudillo, 2016) which, unlike softmax , are sparse (i.e. produce vectors of probability values with components that evaluate to exactly 0 ).”
Learning to Deceive with Attention-Based Explanations (2019, background): “Similar claims about attention’s utility for interpreting models’ predictions are common in the literature (Li et al., 2016; Xu et al., 2015; Choi et al., 2016; Xie et al., 2017; Martins and Astudillo, 2016; Lai and Tan, 2019).”
Scheduled Sampling for Transformers (2019, neutral): “sparsemax mix of the embeddings (Martins and Astudillo, 2016).”
An Attentive Survey of Attention Models (2019, methodology): “Visual attention term was conceived by Mnih et al. [2014] in which attention was proposed for the Image Classification task.”
Learning Classifiers with Fenchel-Young Losses: Generalized Entropies, Margins, and Algorithms (2018, background): “To address these shortcomings, Martins and Astudillo (2016) proposed a new loss based on the projection onto the simplex.”
SparseMAP: Differentiable Sparse Structured Inference (2018, methodology): “By replacing the entropic penalty with a squared `2 norm, Martins & Astudillo (2016) introduced a sparse alternative to softmax, called sparsemax, given by sparsemax(θ) := argmax y∈4d θ>y − 1 2 ‖y‖22 = argmin y∈4d ‖y − θ‖22 .”
Sparse Sequence-to-Sequence Models (2019, methodology): “Our innovations are rooted in the recently proposed sparsemax transformation (Martins and Astudillo, 2016) and FenchelYoung losses (Blondel et al., 2019).”
Deep Neural Networks and Tabular Data: A Survey (2021, methodology): “At each decision step, TabNet first applies a sparse feature mask [112] to perform soft instance-wise feature selection.”
SIGMORPHON 2020 Shared Task 0: Typologically Diverse Morphological Inflection (2020, methodology): “Turkic Turkic aze (azb) Azerbaijani UniMorph Turkic bak Bashkir UniMorph Turkic crh Crimean Tatar UniMorph Turkic kaz Kazakh Nabiyev (2015); Turkicum (2019a) Turkic kir Kyrgyz Aytnatova (2016) Turkic kjh Khakas UniMorph Turkic tuk Turkmen Abdulin (2016); US Embassy (2018) Turkic uig Uyghur Kadeer (2”