Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese

By Shiyu Zhou, Linhao Dong, Shuang Xu, Bo Xu. 2018

130 citations recorded by the source corpus.

Sequence-to-sequence attention-based models have recently shown very promising results on automatic speech recognition (ASR) tasks, which integrate an acoustic, pronunciation and language model into a single neural network. In these models, the Transformer, a new sequence-to-sequence attention-based model relying entirely on self-attention without using RNNs or convolutions, achieves a new single-model state-of-the-art BLEU on neural machine translation (NMT) tasks. Since the outstanding performance of the Transformer, we extend it to speech and concentrate on it as the basic architecture of sequence-to-sequence attention-based model on Mandarin Chinese ASR tasks. Furthermore, we investigate a comparison between syllable based model and context-independent phoneme (CI-phoneme) based model with the Transformer in Mandarin Chinese. Additionally, a greedy cascading decoder with the Transformer is proposed for mapping CI-phoneme sequences and syllable sequences into word sequences. Experiments on HKUST datasets demonstrate that syllable based model with the Transformer performs better than CI-phoneme based counterpart, and achieves a character error rate (CER) of \emph{$28.77\%$}, which is competitive to the state-of-the-art CER of $28.0\%$ by the joint CTC-attention based encoder-decoder network.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

Root Mean Square Layer Normalization (2019, methodology): “Due to its simplicity and requiring no dependencies among training cases, LayerNorm has been widely applied to different neural architectures, which enables remarkable success on various tasks ranging from computer vision [19, 26], speech recognition [37] to natural language processing [31, 35].”
Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss (2020, methodology): “As typically used in sequence-to-sequence transduction tasks [4, 5, 6, 7, 8], Transformerbased models attend over encoder features using decoder features, implying that the decoding has to be done in a label-synchronous way, thereby posing a challenge for streaming speech recognition applications.”
Transformers with convolutional context for ASR (2019, background): “There has been recent research interest in using transformer networks for end-to-end ASR both with CTC loss [13] and in an encoder-decoder framework [14, 15] with modest performance compared to baseline systems.”
A Comparison of Modeling Units in Sequence-to-Sequence Speech Recognition with the Transformer on Mandarin Chinese (2018, methodology): “65 Syllables-D1024-H16 (speed perturb) [9] 28.”
A Comparative Study on Transformer vs RNN in Speech Applications (2019, background): “On the other hand, speech processing tasks have just provided their preliminary results in ASR [2], [11], ST [3] and TTS [4].”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, methodology): “Self-attention and transformer models were also used for ASR, mostly in the sequence-to-sequence architecture [17–19] with notable exceptions of [20,21].”
Close to Human Quality TTS with Transformer (2018, unknown): “Recently, Transformer has been applied in automatic speech recognition (ASR) (Zhou et al. 2018a; Zhou et al. 2018b), proving its ability in acoustic modeling other than natural language process.”
Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding (2019, methodology): “Considering that Transformer model is now the state-of-art model in MT field and also shows a superior performance in ASR filed (Vaswani et al. 2017; Dong, Xu, and Xu 2018; Zhou et al. 2018), we adopt Transformer model as the core structure.”
State-of-the-Art Speech Recognition Using Multi-Stream Self-Attention with Dilated 1D Convolutions (2019, methodology): “With the huge success in NLP, self-attention has been actively investigated for speech recognition as well [5, 6, 7, 8, 9, 10].”
Learn Spelling from Teachers: Transferring Knowledge from Language Models to Sequence-to-Sequence Speech Recognition (2019, methodology): “Experimental setup In this paper, we employ Speech-Transformer [3, 21], a nonrecurrent Seq2Seq model for speech recognition, as the backbone network.”
Enhancing Transformer for End-to-end Speech-to-Text Translation (2019, background): “Some works trained Transformer for (multilingual) ASR with little modiﬁcation to its architecture (Zhou et al., 2018a; Zhou et al., 2018b), showing the feasibility of this approach in terms of results.”
Learning Noise-Invariant Representations for Robust Speech Recognition (2018, methodology): “Over the past several years, a series of papers have developed end-to-end deep learning systems for automatic speech recognition (ASR), advancing the state of the art on a variety of benchmarks [1, 2, 3, 4, 5, 6].”
The MGB-5 Challenge: Recognition and Dialect Identification of Dialectal Arabic Speech (2019, methodology): “Zhengzhou Xinda Institute of Advanced Technology (ZXIAT): ZXIAT submitted two systems, one system using a hybrid HMM/DNN, using a TDNN with trigram LM the other using an end-to-end ASR system based on transformer models [21, 22, 23].”
Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and Language Model Integration (2019, methodology): “, 81, 118 and 581 hours) datasets than existing works [7], [12].”