Pay Less Attention with Lightweight and Dynamic Convolutions

By Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, Michael Auli. 2019

665 citations recorded by the source corpus.

Self-attention is a useful mechanism to build generative models for language and images. It determines the importance of context elements by comparing each element to the current time step. In this paper, we show that a very lightweight convolution can perform competitively to the best reported self-attention results. Next, we introduce dynamic convolutions which are simpler and more efficient than self-attention. We predict separate convolution kernels based solely on the current time-step in order to determine the importance of context elements. The number of operations required by this approach scales linearly in the input length, whereas self-attention is quadratic. Experiments on large-scale machine translation, language modeling and abstractive summarization show that dynamic convolutions improve over strong self-attention models. On the WMT'14 English-German test set dynamic convolutions achieve a new state of the art of 29.7 BLEU.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

TinyBERT: Distilling BERT for Natural Language Understanding (2019, methodology): “Moreover, recent studies (Wu et al., 2019a) also demonstrate that there is redundancy in PLMs.”
Massively Multilingual Neural Machine Translation in the Wild: Findings and Challenges (2019, background): “Finally, while scaling up existing approaches is one way to improve model quality, some approaches or architectures are more efficient to train (Shazeer et al., 2017; Vaswani et al., 2017; Wu et al., 2019), more sample efficient, or faster at inference (Gu et al.”
fairseq: A Fast, Extensible Toolkit for Sequence Modeling (2019, methodology): “, 2015), convolutional models (Gehring et al., 2017; Wu et al., 2019) and Transformer (Vaswani et al.”
Compressive Transformers for Long-Range Sequence Modelling (2019, background): “Wu et al. (2019) show that a convolution-like operator that runs in linear time can actually exceed the performance of the quadratic-time self-attention layer in the Transformer at sentence-to-sentence translation and sentence-level language modelling.”
Beyond English-Centric Multilingual Machine Translation (2020, background): “Recent studies show that encoders are more important for bilingual machine translation (Wu et al., 2019; Kasai et al., 2020), however, these studies are based on systems modeling only a single language direction compared to our setting.”
FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “Other approaches include compressing along the sequence dimension to attend to multiple tokens at once [54, 59, 82, 93].”
Longformer: The Long-Document Transformer (2020, background): “This is analogues to CNNs where stacking layers of small kernels leads to high level features that are built from a large portion of the input (receptive ﬁeld) (Wu et al., 2019).”
Exploring Self-Attention for Image Recognition (2020, background): “Self-attention models have revolutionized machine translation and natural language processing more broadly [33, 37, 7, 6, 40].”
VOLO: Vision Outlooker for Visual Recognition (2021, methodology): “This also makes our model different from the Dynamic Convolution [60] and Involution [34] that generate input-dependent convolution kernels to encode the features.”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “Instead of fixed positional embeddings which encode absolute positional information, we use a convolutional layer similar to [37, 4, 57] which acts as relative positional embedding.”
Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning (2020, neutral): “2 DynamicConv (Wu et al., 2019) 35 MIXER (Ranzato et al.”
SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarization (2019, positive): “• DynamicConv + GPT-2 embeddings with a separator (trained on news + dialogues), • DynamicConv + GPT-2 embeddings (trained on news + dialogues), • Fast Abs RL (trained on dialogues), • Fast Abs RL Enhanced (trained on dialogues), • Transformer (trained on news + dialogues).”
MAESTRO: Matched Speech Text Representations through Modality Matching (2022, methodology): “The Refiner θRefiner includes 2 layers of 8-headed self-attention blocks with 17×1 lightweight convolutions [37].”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “First, we try replacing the depthwise convolution in the convolution module with a lightweight convolution [35], see a significant drop in the performance especially on the devother dataset.”
Knowledge Neurons in Pretrained Transformers (2021, background): “Recently, Wu et al. (2019) and Dong et al. (2021) have pointed out that the feed-forward network module also matters to Transformer.”
Depth-Adaptive Transformer (2019, neutral): “, 2019), summarization (Celikyilmaz et al., 2018; Wu et al., 2019) and natural language understanding (Radford et al.”
BERTScore: Evaluating Text Generation with BERT (2019, background): “Interestingly, the gap between a DynamicConv (Wu et al., 2019) trained on only WMT16 and a Transformer (Ott et al.”