Fixed Encoder Self-Attention Patterns in Transformer-Based Machine Translation

By Alessandro Raganato, Yves Scherrer, Jörg Tiedemann. 2020

97 citations recorded by the source corpus.

Transformer-based models have brought a radical change to neural machine translation. A key feature of the Transformer architecture is the so-called multi-head attention mechanism, which allows the model to focus simultaneously on different parts of the input. However, recent works have shown that most attention heads learn simple, and often redundant, positional patterns. In this paper, we propose to replace all but one attention head of each encoder layer with simple fixed – non-learnable – attentive patterns that are solely based on position and do not require any external knowledge. Our experiments with different data sizes and multiple language pairs show that fixing the attention heads on the encoder side of the Transformer at training time does not impact the translation quality and even increases BLEU scores by up to 3 points in low-resource scenarios.

Subfields: Model Architectures, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

In Machine Translation

Steven C. H. Hoi

Citations in the corpus

ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers (2022, background): “Model compression has been explored from diﬀerent aspects [25, 37, 39, 34, 43, 20, 24, 50, 18, 74, 40, 26, 55, 59, 28, 60, 68, 33, 14, 38, 31].”
Synthesizer: Rethinking Self-Attention for Transformer Models (2020, background): “The most closely related work is ((Raganato et al., 2020)), in which the authors propose us-ing ﬁxed (i.e., not learned) attention patterns in Transformer encoders.”
FNet: Mixing Tokens with Fourier Transforms (2021, background): “(2020) and Raganato et al. (2020) suggest that most connections in the attention sublayer in the encoder - and possibly the decoder - do not need to be learned at all,”
I-BERT: Integer-only BERT Quantization (2021, background): “These techniques can be broadly categorized into: (1) pruning (Fan et al., 2019; Gordon et al., 2020; Han et al., 2015; LeCun et al., 1990; Li et al., 2016b; Mao et al., 2017; 2020; Michel et al., 2019; Molchanov et al., 2016; Raganato et al., 2020; Sanh et al., 2020; Yang et al., 2017); (2) knowled”
GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference (2020, unknown): “Attention head pruning [20, 21] and Encoder unit pruning [22] are examples of this approach.”
Synthesizer: Rethinking Self-Attention in Transformer Models (2020, background): “The most closely related work is [Raganato et al., 2020], in which the authors propose using ﬁxed (i.e., not learned) attention patterns in Transformer encoders.”
ETSformer: Exponential Smoothing Transformers for Time-series Forecasting (2022, background): “(Raganato et al., 2020) replaces attention with fixed, non-learnable positional patterns, obtaining competitive performance on NMT tasks.”