Monotonic Infinite Lookback Attention for Simultaneous Machine Translation
By Naveen Arivazhagan, Colin Cherry, Wolfgang Macherey, Chung-Cheng Chiu, Semih Yavuz, Ruoming Pang, Wei Li, Colin Raffel. 2019
218 citations recorded by the source corpus.
Simultaneous machine translation begins to translate each source sentence before the source speaker is finished speaking, with applications to live and streaming scenarios. Simultaneous systems must carefully schedule their reading of the source sentence to balance quality against latency. We present the first simultaneous translation system to learn an adaptive schedule jointly with a neural machine translation (NMT) model that attends over all source tokens read thus far. We do so by introducing Monotonic Infinite Lookback (MILk) attention, which maintains both a hard, monotonic attention head to schedule the reading of the source sentence, and a soft attention head that extends from the monotonic head back to the beginning of the source. We show that MILk’s adaptive schedule allows it to arrive at latency-quality trade-offs that are favorable to those of a recently proposed wait-k strategy for many latency values.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Model Architectures, Machine Translation
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Computer Vision and Pattern Recognition
- Luc Van Gool
- Dacheng Tao
- Xiaogang Wang
- Ming-Hsuan Yang
- Trevor Darrell
- Jitendra Malik
- Alan L. Yuille
- Andrew Zisserman
In Model Architectures
- Jinyu Li
- Tara N. Sainath
- Shinji Watanabe
- Chung-Cheng Chiu
- Yu Zhang
- Yonghui Wu
- Rohit Prabhavalkar
- Takaaki Hori
In Machine Translation
Citations in the corpus
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “For the simultaneous ST setting, it includes online models with widely used policies: monotonic attention (Raffel et al., 2017), wait-k (Ma et al., 2019), monotonic infinite lookback attention (Arivazhagan et al., 2019b), and mono-tonic multihead attention (Ma et al., 2020b).”
- SIMULEVAL: An Evaluation Toolkit for Simultaneous Translation (2020, background): “…reading the entire text or speech source input, has become an increasingly popular topic for both text and speech translation (Gris-som II et al., 2014; Cho and Esipova, 2016; Gu et al., 2017; Alinejad et al., 2018; Arivazhagan et al., 2019; Zheng et al., 2019; Ma et al., 2020; Ren et al., 2020).”
- Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning (2020, background): “The development of these streaming, end-to-end architectures has seen considerable attention for the tasks of automatic speech recognition [7, 8, 9, 10] and machine translation (MT) [11, 12, 13, 14].”
- FINDINGS OF THE IWSLT 2020 EVALUATION CAMPAIGN (2020, background): “For simplicity, report only the following ones: Flicker is inspired by Arivazhagan et al. (2019a).”
- End-to-End Speech Recognition: A Survey (2023, background): “[40] – the monotonic infinite lookback (MILK) attention”
- A Comparison of End-to-End Models for Long-Form Speech Recognition (2019, methodology): “We explore a few different mechanisms to enforce monotonicity, including the monotonic attention model [15], the monotonic chunkwise attention model [16], the monotonic infinite lookback attention model [17], and the GMM-based monotonic attention model”
- Direct simultaneous speech to speech translation (2021, background): “C L ] 12 J a n 2022 policies (Raffel et al., 2017; Arivazhagan et al., 2019; Ma et al., 2019c) use a closed form estimation on simultaneous alignment during the training time, we find that such estimation can be biased given long sequences, which usually happens in speech applications.”
- Future-Guided Incremental Transformer for Simultaneous Translation (2020, unknown): “Arivazhagan et al. (2019) proposed MILK, which uses a variable based on Bernoulli distribution to determine whether to output.”
- Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “Recently, monotonic infinite lookback (MILK) attention [46] was proposed to attend the entire sequence preceding the trigger point, as shown in Figure 3d.”
- Monotonic Multihead Attention (2019, methodology): “We do not have the tokenizer which processed this data, thus we report tokenized BLEU for IWSLT15 Latency metrics are computed on BPE tokens for WMT15 De-En – consistent with Arivazhagan et al. (2019) – and on word tokens for IWSLT15 En-Vi.”
- Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, methodology): “…have also been made by modifying attention-based models such as monotonic attention (Raffel et al., 2017; Chiu & Raffel, 2017; Fan et al., 2018; Arivazhagan et al., 2019), GMM attention (Graves, 2013; Chiu et al., 2019), triggered attention (TA) (Moritz et al., 2019), Scout Network (Wang et…”
- What the Future Brings: Investigating the Impact of Lookahead for Incremental Neural TTS (2020, unknown): “Several strategies have been proposed to reduce the latency of a sequence-to-sequence model with input text for neural machine translation [9, 10, 11] or incremental speech translation [12, 13, 14].”