Sparse and Constrained Attention for Neural Machine Translation
By Chaitanya Malaviya, Pedro Ferreira, André F. T. Martins. 2018
65 citations recorded by the source corpus.
In neural machine translation, words are sometimes dropped from the source or generated repeatedly in the translation. We explore novel strategies to address the coverage problem that change only the attention transformation. Our approach allocates fertilities to source words, used to bound the attention each word can receive. We experiment with various sparse and constrained attention transformations and propose a new one, constrained sparsemax, shown to be differentiable and sparse. Empirical evaluation is provided in three languages pairs.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Ben Peters
- Vlad Niculae
- Ping Luo
- André F. T. Martins
- Gonçalo M. Correia
- Xiaogang Wang
- Maarten de Rijke
- Marlies van der Wees
In Computer Vision and Pattern Recognition
- Xiaogang Wang
- Ping Luo
- Luc Van Gool
- Bernt Schiele
- Dacheng Tao
- Ming-Hsuan Yang
- Yi Yang
- Trevor Darrell
Citations in the corpus
- Adaptively Sparse Transformers (2019, methodology): “The Transformer architecture (Vaswani et al., 2017) for deep neural networks has quickly risen to prominence in NLP through its efficiency and performance, leading to improvements in the state of the art of Neural Machine Translation (NMT; Junczys-Dowmunt et al., 2018; Ott et al., 2018), as well as i”
- Fine-tune BERT with Sparse Self-Attention Mechanism (2019, methodology): “…formulations have been developed in the literature such as sparsemax (Martins and Astudillo, 2016), fusedmax (Niculae and Blondel, 2017), and constrained sparsemax (Malaviya et al., 2018), they are mostly applied in the classification layer or in the attention-based encoder-decoder architecture.”
- Attention in Natural Language Processing (2019, background): “In [46] and [51], the coverage information is exploited by a constrained distribution function, regulating the amount of attention that the same word receives over time.”
- SSN: Learning Sparse Switchable Normalization via SparsestMax (2019, background): “Another variant constrained sparsemax (Malaviya et al. 2018) produces both sparse andbounded attention weights, yielding a compact and interpretable set of alignments.”
- Understanding Multi-Head Attention in Abstractive Summarization (2019, methodology): “Sparsity has been used to improve the interpretability of single-head attention architectures (Malaviya et al., 2018; Deng et al., 2018; Niculae et al., 2018).”
- Sparse Sequence-to-Sequence Models (2019, background): “Combining sparse attention with fertility constraints has been recently proposed by Malaviya et al. (2018).”
- SMYRF: Efficient Attention using Asymmetric Clustering (2020, background): “For example, in some variants [17, 27] sparsification of the attention map happens after instantiating the matrix, and so quadratic memory is still used before instantiation.”