Faster Transformer Decoding: N-gram Masked Self-Attention
By Ciprian Chelba, M. Chen, Ankur Bapna, Noam Shazeer. 2020
20 citations recorded by the source corpus.
Motivated by the fact that most of the information relevant to the prediction of target tokens is drawn from the source sentence $S=s_1, \ldots, s_S$, we propose truncating the target-side window used for computing self-attention by making an $N$-gram assumption. Experiments on WMT EnDe and EnFr data sets show that the $N$-gram masked self-attention model loses very little in BLEU score for $N$ values in the range $4, \ldots, 8$, depending on the task.
Subfields: Artificial Intelligence, Natural Language Processing, Efficiency & Compression
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Noah A. Smith
- Krzysztof Marcin Choromanski
- Jungo Kasai
- Adrian Weller
- Hao Peng
- Tri Dao
- Mohit Bansal
- James Cross
In Natural Language Processing
- Noah A. Smith
- Jungo Kasai
- Nikolaos Pappas
- Hao Peng
- James Cross
- Graham Neubig
- Mohit Bansal
- Percy Liang
In Efficiency & Compression
- Noah A. Smith
- Nikolaos Pappas
- Adrian Weller
- Valerii Likhosherstov
- Krzysztof Marcin Choromanski
- Jared Davis
- Xingyou Song
- James Cross
Citations in the corpus
- Rethinking Attention with Performers (2020, background): “…(Roy et al., 2020) which applies k -means clustering to learn dynamic sparse attention regions, or (Kitaev et al., 2020), where locality sensitive hashing is used to group together tokens of similar embeddings), sliding windows (Beltagy et al., 2020), or truncated targeting (Chelba et al., 2020).”
- Masked Language Modeling for Proteins via Linearly Scalable Long-Context Transformers (2020, background): “[42] which applies k-means clustering to learn dynamic sparse attention regions, or [29], where locality sensitive hashing is used to group together tokens of similar embeddings), sliding windows [2], or truncated targeting [5].”
- Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, methodology): “ed its speed and accuracy with non-autoregressive models. Other works proposed methods to reduce attention computation in autoregressive transformers (Kitaev et al., 2020; Katharopoulos et al., 2020; Chelba et al., 2020; Peng et al., 2021). Some of these methods can be used orthogonally to further f”