Online and Linear-Time Attention by Enforcing Monotonic Alignments

By Colin Raffel, Minh-Thang Luong, Peter J. Liu, Ron J. Weiss, Douglas Eck. 2017

277 citations recorded by the source corpus.

Recurrent neural network models with an attention mechanism have proven to be extremely effective on a wide variety of sequence-to-sequence problems. However, the fact that soft attention mechanisms perform a pass over the entire input sequence when producing each element in the output sequence precludes their use in online settings and results in a quadratic time complexity. Based on the insight that the alignment between input and output sequence elements is monotonic in many problems of interest, we propose an end-to-end differentiable method for learning monotonic alignments which, at test time, enables computing attention online and in linear time. We validate our approach on sentence summarization, machine translation, and online speech recognition problems and achieve results competitive with existing sequence-to-sequence models.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Model Architectures

Citations in the corpus

WT5?! Training Text-to-Text Models to Explain their Predictions (2020, methodology): “…some means of interpretability “for free” by examining the weight assigned by the neural network to different regions in the input (Graves, 2013; Raffel et al., 2017; Huang et al., 2018), but this introspection method has been shown to be unreliable (Jain & Wallace, 2019; Serrano & Smith,…”
Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “For the simultaneous ST setting, it includes online models with widely used policies: monotonic attention (Raffel et al., 2017), wait-k (Ma et al., 2019), monotonic inﬁnite lookback attention (Arivazhagan et al., 2019b), and mono-tonic multihead attention (Ma et al., 2020b).”
Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding (2022, methodology): “Imagen explores pretrained text encoders: BERT [15], T5 [53] and CLIP [48].”
State-of-the-Art Speech Recognition with Sequence-to-Sequence Models (2017, background): “[4] C. Raffel, M. Luong, P. J. Liu, R.J. Weiss, and D. Eck, “Online and Linear-Time Attention by Enforcing Monotonic Alignments,” in Proc.”
A Survey on Neural Speech Synthesis (2021, background): “For monotonic attention [282, 46, 104, 404, 341], the attention position is monotonically increasing, which also leverages the prior that the alignments between text and speech are monotonic.”
Deep Voice 3: 2000-Speaker Neural Text-to-Speech (2017, background): “Recent improvements in attention mechanisms relevant to Deep Voice 3 include enforced-monotonic attention during training (Raffel et al., 2017), fully-attentional non-recurrent architectures (Vaswani et al., 2017), and convolutional sequence-to-sequence models (Gehring et al., 2017).”
End-to-End Adversarial Text-to-Speech (2020, neutral): “Several mechanisms have been proposed to exploit monotonicity in tasks that require sequence alignment, including attention mechanisms [11, 20, 24, 48, 62, 72], loss functions [19, 21] and search-based approaches [29].”
Location-Relative Attention Mechanisms for Robust Long-Form Speech Synthesis (2019, background): “The hybrid mechanism still has occasional alignment issues which led a number of authors to develop attention mechanisms that directly exploit monotonicity [9, 5, 6].”
Monotonic Chunkwise Attention (2017, methodology): “We use the model of (Raffel et al., 2017), which is itself based on that of (Zhang et al., 2016).”
Exploring neural transducers for end-to-end speech recognition (2017, background): “Alternatives have been proposed to circumvent this issue [21, 1] and build attention models with monotonic attention and streaming decoders, but none of them are able to completely match the performance of the full attention models.”
End-to-end Code-switched TTS with Mix of Monolingual Recordings (2019, methodology): “We use monotonic attention, which achieves soft-monotonic attention by training in expectation [19].”
Revisiting Character-Based Neural Machine Translation with Capacity and Compression (2018, background): “Examples of such strategies include sparse attention (Raffel et al., 2017) and discrete auto-encoders (Kaiser et al.”
Deep Voice 3: Scaling Text-to-Speech with Convolutional Sequence Learning (2017, background): “Recent improvements in attention mechanisms relevant to Deep Voice 3 include enforced-monotonic attention during training (Raffel et al., 2017), fully-attentional non-recurrent architectures (Vaswani et al., 2017), and convolutional sequenceto-sequence models (Gehring et al., 2017).”
Self-attention Aligner: A Latency-control End-to-end Model for ASR Using Self-attention Network and Chunk-hopping (2019, methodology): “End-to-end models [1, 2, 3, 4, 5] have greatly simplified the automatic speech recognition (ASR) system by combining acoustic model, language model and an acoustic-to-text alignment mechanism (e.”
Morphological Inflection Generation with Hard Monotonic Attention (2016, background): “A recent work by Raffel et al. (2017) jointly learns the hard monotonic alignments and transduction while maintaining the dependency between the encoder and the decoder.”