Monotonic Multihead Attention

By Xutai Ma, Juan Pino, James Cross, Liezl Puzon, Jiatao Gu. 2019

152 citations recorded by the source corpus.

Simultaneous machine translation models start generating a target sequence before they have encoded or read the source sequence. Recent approaches for this task either apply a fixed policy on a state-of-the art Transformer model, or a learnable monotonic attention on a weaker recurrent neural network-based structure. In this paper, we propose a new attention mechanism, Monotonic Multihead Attention (MMA), which extends the monotonic attention mechanism to multihead attention. We also introduce two novel and interpretable approaches for latency control that are specifically designed for multiple attentions heads. We apply MMA to the simultaneous machine translation task and demonstrate better latency-quality tradeoffs compared to MILk, the previous state-of-the-art approach. We also analyze how the latency controls affect the attention span and we motivate the introduction of our model by analyzing the effect of the number of decoder layers and heads on quality and latency.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Juan Pino
2. Changhan Wang
3. Jiatao Gu
4. Ann Lee
5. Wei-Ning Hsu
6. Peng-Jen Chen
7. Yun Tang
8. Hongyu Gong

In Computer Vision and Pattern Recognition

1. Jiatao Gu
2. Juan Pino
3. Philipp Koehn
4. Changhan Wang
5. Anne Wu
6. Xutai Ma
7. Kenneth Heafield
8. Peng-Jen Chen

In Model Architectures

1. Dmytro Okhonko
2. Juan Pino
3. Ann Lee
4. Kenneth Heafield
5. Philipp Koehn
6. Hongyu Gong
7. Changhan Wang
8. Anne Wu

Citations in the corpus

• Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “For the simultaneous ST setting, it includes online models with widely used policies: monotonic attention (Raffel et al., 2017), wait-k (Ma et al., 2019), monotonic infinite lookback attention (Arivazhagan et al., 2019b), and mono-tonic multihead attention (Ma et al., 2020b).”
• SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation (2020, background): “…reading the entire text or speech source input, has become an increasingly popular topic for both text and speech translation (Gris-som II et al., 2014; Cho and Esipova, 2016; Gu et al., 2017; Alinejad et al., 2018; Arivazhagan et al., 2019; Zheng et al., 2019; Ma et al., 2020; Ren et al., 2020).”
• What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTS (2020, unknown): “Several strategies have been proposed to reduce the latency of a sequence-to-sequence model with input text for neural machine translation [9, 10, 11] or incremental speech translation [12, 13, 14].”
• Fast Transformers with Clustered Attention (2020, background): “Recently, some of these approaches have also been applied in the context of transformers [17].”
• Future-Guided Incremental Transformer for Simultaneous Translation (2020, unknown): “Most of the previous methods use the unidirectional encoder (Arivazhagan et al. 2019; Ma et al. 2020) or fune-tuning a trained model (Dalvi et al. 2018) to reduce the computational cost.”
• Direct simultaneous speech to speech translation (2021, methodology): “tonic multihead attention (MMA) is a variant of multihead attention, enabling the model in partial input processing and incremental decoding (Ma et al., 2019c).”