Accelerating Neural Transformer via an Average Attention Network

By Biao Zhang, Deyi Xiong, Jinsong Su. 2018

128 citations recorded by the source corpus.

With parallelizable attention networks, the neural Transformer is very fast to train. However, due to the auto-regressive architecture and self-attention in the decoder, the decoding procedure becomes slow. To alleviate this issue, we propose an average attention network as an alternative to the self-attention network in the decoder of the neural Transformer. The average attention network consists of two layers, with an average layer that models dependencies on previous positions and a gating layer that is stacked over the average layer to enhance the expressiveness of the proposed attention network. We apply this network on the decoder part of the neural Transformer to replace the original target-side self-attention model. With masking tricks and dynamic programming, our model enables the neural Transformer to decode sentences over four times faster than its original version with almost no loss in training time and translation performance. We conduct a series of experiments on WMT17 translation tasks, where on 6 different language pairs, we obtain robust and consistent speed-ups in decoding.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Efficiency & Compression

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Model Architectures

In Efficiency & Compression

Citations in the corpus

Fast Transformer Decoding: One Write-Head is All You Need (2019, background): “Another is to reduce the number of positions being attended-to, either by attending to a local neighborhood, or by otherwise compressing the number of memory positions, as in [Liu et al., 2018], [Zhang et al., 2018], [Povey et al., 2018].”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “For comparison, we re-implemented averaged attention networks (AAN; Zhang et al. 2018) which compute a uniform average over past model states instead of a weighted average as in self-attention.”
Training with Quantization Noise for Extreme Model Compression (2020, methodology): “Others have worked on lightweight architectures, by modifying existing models [24, 25, 26] or developing new networks, such as MobileNet [16], ShuffleNet [15], and EfficientNet [4] in vision.”
HAT: Hardware-Aware Transformers for Efficient Natural Language Processing (2020, background): “Zhang et al. (2018) and Kim et al. (2019) also proposed AAN and SSRU to replace the attention mechanism.”
Self-Training for End-to-End Speech Translation (2020, background): “A bottleneck linear layer with dimension 128 is inserted prior to the softmax over the target vocabulary and the decoder is an average attention network [30].”
A Survey of Transformers (2021, methodology): “[164] design an efficient Transformer decoder variant called average attention network that uses a discrete uniform distribution as the sole source of attention distribution.”
Faster Transformer Decoding: N-gram Masked Self-Attention (2020, unknown): “The results in (Zhang et al., 2018) do show that optimizing the self-attention computation can have a signiﬁcant impact on decoding speed.”
TRAR: Routing the Attention Spans in Transformer for Visual Question Answering (2021, background): “After gaining dominance in the field of natural language processing [60, 10, 74, 9, 27], Transformer [60] also becomes the prime choice of many vision-and-language (V&L) tasks [14, 7, 30].”
Deep Encoder, Shallow Decoder: Reevaluating the Speed-Quality Tradeoff in Machine Translation (2020, methodology): “Zhang et al. (2018) used dynamic programming in an average attention network to accelerate inference.”
A Lightweight Recurrent Network for Sequence Modeling (2019, methodology): “The design of the input gate i t and forget gate f t is inspired by the twin-gate mechanism in ATR (Zhang et al., 2018).”
The Cascade Transformer: an Application for Efficient Answer Sentence Selection (2020, methodology): “Being aware of this problem, the research community has been developing techniques to prune unnecessary network parameters (Lan et al., 2019; Sanh et al., 2019) or optimize the transformer architecture (Zhang et al., 2018; Xiao et al., 2019).”
Large-Scale Transfer Learning for Natural Language Generation (2019, methodology): “Table 1 shows a typical dialog from Per-sonaChat (Zhang et al., 2018b), one of the largest multi-turn open-domain dialog dataset available.”
Future-Guided Incremental Transformer for Simultaneous Translation (2020, methodology): “To avoid the high complexity caused by recalculation of the consumed source hidden states, inspired by Zhang, Xiong, and Su (2018), we propose the incremental Transformer, including a unidirectional encoder and a decoder with an average embedding layer.”
Gender bias amplification during Speed-Quality optimization in Neural Machine Translation (2021, unknown): “Average Attention Networks (AAN): Average Attention Networks reduce the quadratic complexity of the decoder attention mechanism to linear time by replacing the decoder-side self-attention with an average-attention operation using a ﬁxed weight for all time-steps (Zhang et al., 2018).”
Balancing Cost and Benefit with Tied-Multi Transformers (2020, background): “, 2016) and average attention networks (Xiong et al., 2018) address this topic.”