A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese

By Shiyu Zhou, Linhao Dong, Shuang Xu, Bo Xu. 2018

69 citations recorded by the source corpus.

The choice of modeling units is critical to automatic speech recognition (ASR) tasks. Conventional ASR systems typically choose context-dependent states (CD-states) or context-dependent phonemes (CD-phonemes) as their modeling units. However, it has been challenged by sequence-to-sequence attention-based models, which integrate an acoustic, pronunciation and language model into a single neural network. On English ASR tasks, previous attempts have already shown that the modeling unit of graphemes can outperform that of phonemes by sequence-to-sequence attention-based model. In this paper, we are concerned with modeling units on Mandarin Chinese ASR tasks using sequence-to-sequence attention-based models with the Transformer. Five modeling units are explored including context-independent phonemes (CI-phonemes), syllables, words, sub-words and characters. Experiments on HKUST datasets demonstrate that the lexicon free modeling units can outperform lexicon related modeling units in terms of character error rate (CER). Among five modeling units, character based model performs best and establishes a new state-of-the-art CER of $26.64\%$ on HKUST datasets without a hand-designed lexicon and an extra language model integration, which corresponds to a $4.8\%$ relative improvement over the existing best CER of $28.0\%$ by the joint CTC-attention based encoder-decoder network.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Jinyu Li
5. Sheng Zhao
6. Dong Yu
7. Shinji Watanabe
8. Yanqing Liu

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Tao Qin
4. Tie-Yan Liu
5. Jinyu Li
6. Yanqing Liu
7. Yi Ren
8. Rohit Prabhavalkar

Citations in the corpus

• End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, neutral): “While many advances in end-to-end ASR come as the result of neural architecture search (Prabhavalkar et al., 2017; Zhou et al., 2018; Chiu et al., 2018b), we additionally show that simple semi-supervision via pseudo-labeling significantly bridges the performance gap between a variety of different mo”
• Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, background): “Originally proposed as a better replacement of Recurrent Neural Network (RNN) for neural machine translation, Transformer based models got a lot of traction in the field of speech recognition recently [11, 12, 13, 14].”
• Neural Speech Synthesis with Transformer Network (2018, methodology): “Recently, Transformer has been applied in automatic speech recognition (ASR) (Zhou et al. 2018a; 2018b), proving its ability in acoustic modeling other than natural language process.”
• Token-Level Ensemble Distillation for Grapheme-to-Phoneme Conversion (2019, methodology): “Besides, we adopt Transformer [9] instead of RNN or CNN as the basic encoder-decoder model structure, since it demonstrates advantages in a variety of sequence to sequence tasks, such as neural machine translation [9], text summarization [10], automatic speech recognition [11].”
• Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, background): “In previous work [17, 18], it has been empirically shown that the grapheme-based encoder-decoder models outperform the phoneme-based approach, while [17] find that use of lexica is still useful for recognizing rare words such as named entities.”
• Self-attention Aligner: A Latency-control End-to-end Model for ASR Using Self-attention Network and Chunk-hopping (2019, methodology): “Table 5 lists the comparison between the SAA model and other published models [20, 11, 14, 21] on the HKUST dataset.”
• Close to Human Quality TTS with Transformer (2018, unknown): “Recently, Transformer has been applied in automatic speech recognition (ASR) (Zhou et al. 2018a; Zhou et al. 2018b), proving its ability in acoustic modeling other than natural language process.”
• The MGB-5 Challenge: Recognition and Dialect Identification of Dialectal Arabic Speech (2019, methodology): “Zhengzhou Xinda Institute of Advanced Technology (ZXIAT): ZXIAT submitted two systems, one system using a hybrid HMM/DNN, using a TDNN with trigram LM the other using an end-to-end ASR system based on transformer models [21, 22, 23].”
• Hybrid Autoregressive Transducer (HAT) (2020, background): “This has led some to conclude a lexicon or external LM is unnecessary [23, 24, 25].”
• Learning Noise-Invariant Representations for Robust Speech Recognition (2018, methodology): “Typically these models consist of either Recurrent Neural Networks (RNNs) with Sequence-to-Sequence (Seq2Seq) architectures [7] and attention mechanisms [8, 9], RNN transducers [10], transformer networks [11, 6], convolutional neural networks paired with transformer networks [12, 13], or RNNs traine”