Deep-FSMN for Large Vocabulary Continuous Speech Recognition

By Shiliang Zhang, Ming Lei, Zhijie Yan, Lirong Dai. 2018

129 citations recorded by the source corpus.

In this paper, we present an improved feedforward sequential memory networks (FSMN) architecture, namely Deep-FSMN (DFSMN), by introducing skip connections between memory blocks in adjacent layers. These skip connections enable the information flow across different layers and thus alleviate the gradient vanishing problem when building very deep structure. As a result, DFSMN significantly benefits from these skip connections and deep structure. We have compared the performance of DFSMN to BLSTM both with and without lower frame rate (LFR) on several large speech recognition tasks, including English and Mandarin. Experimental results shown that DFSMN can consistently outperform BLSTM with dramatic gain, especially trained with LFR using CD-Phone as modeling units. In the 20000 hours Fisher (FSH) task, the proposed DFSMN can achieve a word error rate of 9.4% by purely using the cross-entropy criterion and decoding with a 3-gram language model, which achieves a 1.5% absolute improvement compared to the BLSTM. In a 20000 hours Mandarin recognition task, the LFR trained DFSMN can achieve more than 20% relative improvement compared to the LFR trained BLSTM. Moreover, we can easily design the lookahead filter order of the memory blocks in DFSMN to control the latency for real-time applications.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Yi Ren
6. Zhou Zhao
7. Sheng Zhao
8. Dong Yu

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Zhou Zhao
6. Frank K. Soong
7. Sheng Zhao
8. Jinglin Liu

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “DeviceTTS [123] leverages the model structure of DFSMN [434] and mix-resolution decoder to predict multiple frames in one decoding step to speed up inference.”
• Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, methodology): “TTS and ASR TTS (Arik et al., 2017; Wang et al., 2017; Shen et al., 2018; Ping et al., 2019) and ASR (Chiu et al., 2018; Zhang et al., 2018; Xiong et al., 2018; Yang et al., 2018) (Tjandra et al., 2017; 2018) relied on two well-trained ASR and TTS models to further boost the accuracy with unpaired…”
• SpeechMoE: Scaling to Large Acoustic Models with Dynamic Routing Mixture of Experts (2021, methodology): “Our acoustic models consist of four components: MoE layer, sequential memory layer [26], self-attention layer [27] and the output softmax layer.”
• DeviceTTS: A Small-Footprint, Fast, Stable Network for On-Device Text-to-Speech (2020, unknown): “The basic component of DeviceTTS is Deep Feedforward Sequential Memory Network (DFSMN) [14], which is a feedforward neural network with strong modeling ability but small model size.”
• FRCRN: Boosting Feature Representation Using Frequency Recurrence for Monaural Speech Enhancement (2022, unknown): “The reason that we apply FSMN instead of using LSTM is because that FSMN not only achieves competitive performance compared to LSTM, but also requires only about a quarter of the parameters of LSTM [9].”
• Deep Feed-Forward Sequential Memory Networks for Speech Synthesis (2018, unknown): “We give an overview of Deep-FSMN (DFSMN) [24] in this section.”
• Simplified Self-Attention for Transformer-Based end-to-end Speech Recognition (2020, methodology): “In this paper, we validated the proposed SSAN-based transformer on three Mandarin speech recognition datasets: public AISHELL-1 corpus [24], internal 1000 and 20,000 hours corpus same as used in [25].”