Monotonic Chunkwise Attention

By Chung-Cheng Chiu, Colin Raffel. 2017

275 citations recorded by the source corpus.

Sequence-to-sequence models with soft attention have been successfully applied to a wide variety of problems, but their decoding process incurs a quadratic time and space cost and is inapplicable to real-time sequence transduction. To address these issues, we propose Monotonic Chunkwise Attention (MoChA), which adaptively splits the input sequence into small chunks over which soft attention is computed. We show that models utilizing MoChA can be trained efficiently with standard backpropagation while allowing online and linear-time decoding at test time. When applied to online speech recognition, we obtain state-of-the-art results and match the performance of a model using an offline soft attention mechanism. In document summarization experiments where we do not expect monotonic alignments, we show significantly improved performance compared to a baseline monotonic attention-based model.

Subfields: Artificial Intelligence, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Yu Zhang
3. Quoc V. Le
4. Tao Qin
5. Tie-Yan Liu
6. Dong Yu
7. Tara N. Sainath
8. Yonghui Wu

In Model Architectures

1. Jinyu Li
2. Tara N. Sainath
3. Shinji Watanabe
4. Tie-Yan Liu
5. Tao Qin
6. Yonghui Wu
7. Chung-Cheng Chiu
8. Yu Zhang

Citations in the corpus

• Generating Long Sequences with Sparse Transformers (2019, methodology): “Outside of generative modeling, there are several works relevant to improving the efficiency of attention based off chunking (Chiu & Raffel, 2017) or using fixed length representations (Britz et al., 2017).”
• Efficient Content-Based Sparse Attention with Routing Transformers (2020, background): “, 2015), while (Chiu and Raffel, 2017) dynamically segment the sequence into variable sized-chunks.”
• Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling (2019, background): “To date, this framework has produced a number of state-of-the-art results in machine translation [2, 21, 23], speech recognition [3, 4, 5, 10, 11, 12, 13, 14, 16, 15, 20, 22], speech synthesis [6, 8, 19], and speech translation [7, 9].”
• A Survey on Neural Speech Synthesis (2021, background): “For monotonic attention [282, 46, 104, 404, 341], the attention position is monotonically increasing, which also leverages the prior that the alignments between text and speech are monotonic.”
• End-to-End Adversarial Text-to-Speech (2020, neutral): “Several mechanisms have been proposed to exploit monotonicity in tasks that require sequence alignment, including attention mechanisms [11, 20, 24, 48, 62, 72], loss functions [19, 21] and search-based approaches [29].”
• WeNet: Production Oriented Streaming and Non-Streaming End-to-End Speech Recognition Toolkit (2021, background): “Either great effort is required or obvious accuracy loss is introduced to make such model work in the streaming fashion [16, 17, 18].”
• Monotonic Infinite Lookback Attention for Simultaneous Machine Translation (2019, background): “To address this issue, Chiu and Raffel (2018) proposed monotonic chunkwise attention (MoChA), which allows the model to perform soft attention over a small fixed-length chunk preceding ti, i.e. over all available encoder states, hti−cs+1, hti−cs+2, . . . , hti for some fixed chunk size cs .”
• Direct simultaneous speech to speech translation (2021, background): “…training: While (Raffel et al., 2017) assume a hard alignment between encoder and decoder (only one encoder state is passed to the decoder at any time), Chiu & Raffel (2018) introduce monotonic chunkwise attention (MoChA), which enables soft attention in a chunk following the moving attention head.”
• Monotonic Multihead Attention (2019, background): “To address these issues, Chiu & Raffel (2018) introduce Monotonic Chunkwise Attention (MoChA), which allows the decoder to apply softmax attention over a chunk (subsequence of encoder positions).”
• Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, methodology): “To support streaming modeling in E2E ASR models, various efforts have also been made by modifying attention-based models such as monotonic attention (Raffel et al., 2017; Chiu & Raffel, 2017; Fan et al., 2018; Arivazhagan et al., 2019), GMM attention (Graves, 2013; Chiu et al., 2019), triggered…”
• Recognizing Long-Form Speech Using Streaming End-to-End Models (2019, result): “2In the authors’ experience, RNN-T models are better suited for streaming long-form ASR tasks, compared to other E2E variants [20, 21, 22].”
• Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning (2020, methodology): “[20, 21, 22] use encoder-decoder architectures with attention, but compute the attention alignments 1”
• Towards Online End-to-end Transformer Automatic Speech Recognition (2019, background): “Monotonic chunkwise attention (MoChA) uses a trainable monotonic energy function to shift the chunk window [27].”
• s-Transformer: Segment-Transformer for Robust Neural Speech Synthesis (2020, background): “Various attention mechanisms are tried in the encoder-decoder attention model, like stepwise monotonic attention [10], forward-Samples are available at https://vancycici.github.io/sTransformer/ attention[11], monotonic chunkwise attention [12], and etc.”
• Neural Data-to-Text Generation via Jointly Learning the Segmentation and Correspondence (2020, methodology): “Our model, similar to chunkwise attention (Chiu and Raffel, 2018) or coarse-to-fine attention (Ling and Rush, 2017), reduces the cost to O(M +N), where we select the record in O(M) at the beginning of each segment and attend only to the selected record in O(N) when decoding every word.”