Convolutional Self-Attention Network
By Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, Zhaopeng Tu. 2018
11 citations recorded by the source corpus.
Self-attention network (SAN) has recently attracted increasing interest due to its fully parallelized computation and flexibility in modeling dependencies. It can be further enhanced with multi-headed attention mechanism by allowing the model to jointly attend to information from different representation subspaces at different positions (Vaswani et al., 2017). In this work, we propose a novel convolutional self-attention network (CSAN), which offers SAN the abilities to 1) capture neighboring dependencies, and 2) model the interaction between multiple attention heads. Experimental results on WMT14 English-to-German translation task demonstrate that the proposed approach outperforms both the strong Transformer baseline and other existing works on enhancing the locality of SAN. Comparing with previous work, our model does not introduce any new parameters.
Subfields: Model Architectures
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Model Architectures
- Quoc V. Le
- Irwan Bello
- Barret Zoph
- Jonathon Shlens
- Frank Hutter
- Mingxing Tan
- Ashish Vaswani
- Alan L. Yuille
Citations in the corpus
- Attention Augmented Convolutional Networks (2019, background): “Using self-attention in cooperation with convolutions is a theme shared by recent work in Natural Language Processing [49] and Reinforcement Learning [52].”
- On the localness modeling for the self-attention based end-to-end speech synthesis (2020, background): “Besides, local contexts generally play an important role in sequential modeling (Shaw, Uszkoreit, & Vaswani, 2018; Yang et al., 2019; Yang, Tu, Wong, Meng, et al., 2018), but self-attention conducts weighed averaging on the whole sentence, which may lead to the dispersion of the attention…”