Semantic Mask for Transformer based End-to-End Speech Recognition

By Chengyi Wang, Yu Wu, Yujiao Du, Jinyu Li, Shujie Liu, Liang Lu, Shuo Ren, Guoli Ye, Sheng Zhao, Ming Zhou. 2019

53 citations recorded by the source corpus.

Attention-based encoder-decoder model has achieved impressive results for both automatic speech recognition (ASR) and text-to-speech (TTS) tasks. This approach takes advantage of the memorization capacity of neural networks to learn the mapping from the input sequence to the output sequence from scratch, without the assumption of prior knowledge such as the alignments. However, this model is prone to overfitting, especially when the amount of training data is limited. Inspired by SpecAugment and BERT, in this paper, we propose a semantic mask based regularization for training such kind of end-to-end (E2E) model. The idea is to mask the input features corresponding to a particular output token, e.g., a word or a word-piece, in order to encourage the model to fill the token based on the contextual information. While this approach is applicable to the encoder-decoder framework with any type of neural network architecture, we study the transformer-based model for ASR in this work. We perform experiments on Librispeech 960h and TedLium2 data sets, and achieve the state-of-the-art performance on the test set in the scope of E2E models.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Zhuo Chen
3. Takuya Yoshioka
4. Shujie Liu
5. Yu Wu
6. Sanyuan Chen
7. Jian Wu
8. Naoyuki Kanda

Citations in the corpus

• Investigation of Practical Aspects of Single Channel Speech Separation for ASR (2021, methodology): “We can also see that, even though the separation model is tuned with ASRmatched, the consistent performance improvement still exists when the ASR model is changed to ASR [38] which uses a slightly different feature and architecture.”
• MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation (2020, unknown): “Thus, we introduce MAM in this section. et al., 2019; Wang et al., 2019, 2020b) which rely on transcription to segment the speech audio with forced alignment tools(Povey et al., 2011; McAuliffe et al., 2017).”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “7 Semantic mask [327] 2019 AED Transformer 2.”
• Don’t Shoot Butterfly with Rifles: Multi-Channel Continuous Speech Separation with Early Exit Transformer (2020, methodology): “For example, for automatic speech recognition (ASR) tasks, a common setting is to use twelve [15] or more layers [16] in the encoder.”
• Ultra Fast Speech Separation Model with Teacher Student Learning (2021, methodology): “Two numbers in a cell denote %WER of the hybrid ASR model used in LibriCSS [10] and E2E Transformer based ASR model [27].”