Convolutional Self-Attention Networks

By Baosong Yang, Longyue Wang, Derek F. Wong, Lidia S. Chao, Zhaopeng Tu. 2019

135 citations recorded by the source corpus.

Self-attention networks (SANs) have drawn increasing interest due to their high parallelization in computation and flexibility in modeling dependencies. SANs can be further enhanced with multi-head attention by allowing the model to attend to information from different representation subspaces. In this work, we propose novel convolutional self-attention networks, which offer SANs the abilities to 1) strengthen dependencies among neighboring elements, and 2) model the interaction between features extracted by multiple attention heads. Experimental results of machine translation on different language pairs and model settings show that our approach outperforms both the strong Transformer baseline and other existing models on enhancing the locality of SANs. Comparing with prior studies, the proposed model is parameter free in terms of introducing no more parameters.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Quoc V. Le
2. Yu Zhang
3. Yonghui Wu
4. Jinyu Li
5. Ruoming Pang
6. Chunhua Shen
7. Chung-Cheng Chiu
8. Mohit Bansal

In Computer Vision and Pattern Recognition

1. Chunhua Shen
2. Luc Van Gool
3. Dacheng Tao
4. Quoc V. Le
5. Alan L. Yuille
6. Xiaogang Wang
7. Ming-Hsuan Yang
8. Trevor Darrell

In Model Architectures

1. Quoc V. Le
2. Irwan Bello
3. Barret Zoph
4. Jiahui Yu
5. Jonathon Shlens
6. Chunhua Shen
7. Xiangxiang Chu
8. Ruoming Pang

Citations in the corpus

• Attention Augmented Convolutional Networks (2019, background): “Using self-attention in cooperation with convolutions is a theme shared by recent work in Natural Language Processing [49] and Reinforcement Learning [52].”
• Conformer: Convolution-augmented Transformer for Speech Recognition (2020, background): “Concurrently, papers like [15, 16] have augmented self-attention with relative position based information that maintains equivariance.”
• Conditional Positional Encodings for Vision Transformers (2021, methodology): “Convolutions are used to model local relations in ASR and machine translation (Gulati et al., 2020; Mohamed et al., 2019; Yang et al., 2019a; Yu et al., 2018).”
• On the use of BERT for Neural Machine Translation (2019, result): “Using embedding mostly influenced by neighboring words seems to echo the recent results of convolutional self attention network (Yang et al., 2019).”