Improving Transformer-based Speech Recognition Using Unsupervised Pre-training

By Dongwei Jiang, Xiaoning Lei, Wubo Li, Ne Luo, Yuxuan Hu, Wei Zou, Xiangang Li. 2019

106 citations recorded by the source corpus.

Speech recognition technologies are gaining enormous popularity in various industrial applications. However, building a good speech recognition system usually requires large amounts of transcribed data, which is expensive to collect. To tackle this problem, an unsupervised pre-training method called Masked Predictive Coding is proposed, which can be applied for unsupervised pre-training with Transformer based model. Experiments on HKUST show that using the same training data, we can achieve CER 23.3%, exceeding the best end-to-end model by over 0.2% absolute CER. With more pre-training data, we can further reduce the CER to 21.0%, or a 11.8% relative CER reduction over baseline.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Recognition (2021, methodology): “More recently, Transformer architectures have been adopted for self-supervised pre-training (Jiang et al., 2019; Baevski et al., 2020c).”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Another line of work is pre-training representations on unlabeled speech followed by ﬁne-tuning on labeled data [16, 17, 18, 19, 20, 21, 22, 23, 24].”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “In [24], [26], BERT-style masked reconstruction following the standard BERT masking policy is employed to pre-train”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “In speech, instead of prediction, we randomly mask temporal slices of acoustic features and attempt to reconstruct them [21, 22, 23, 24, 25, 16].”
Speech Bert Embedding for Improving Prosody in Neural TTS (2021, methodology): “pre-training is a widely used strategy in natural language processing and automatic speech recognition (ASR) tasks, among which Bidirectional Encoder Representations from Transformers (BERT) has been proven successful [12, 13, 14, 15].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “We use the following abbreviations: EN : English; Multi : multilingual; ZH : Chinese; Fr : French; ASR : automatic speech recognition; PR : phoneme recognition; PC : phoneme classification; SID : speaker identification; ASV : automatic speaker verification; Sentiment : sentiment analysis; ST : speec”
Didispeech: A Large Scale Mandarin Speech Corpus (2020, methodology): “However, the LibriTTS is a subset of the LibriSpeech [24] database which is designed for ASR tasks.”
Improved Speech Representations with Multi-Target Autoregressive Predictive Coding (2020, background): “Unsupervised speech representation learning, which aims to learn a function that transforms surface features, such as audio waveforms or spectrograms, to higher-level representations using only unlabeled speech, has received great attention recently (Baevski et al., 2019, 2020; Liu et al., 2020; Son”
Audio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation (2020, methodology): “The pretraining model learns the robust speech representations for speech processing tasks, for example, ASR and speaker recognition, with the self-supervised learning approaches [4, 5, 6, 7, 8].”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “For zero-masking (Jiang et al. 2019; Liu et al. 2020; Wang, Tang, and Livescu 2020; Chi et al. 2021; Ling and Liu 2020), we have c1:T = f(x1:T ◦m1:T ) (9) x̂t = g(ct) (10) Lt = ‖x̂t − xt‖1 , (11) where the ◦ operator denotes the Hadamard product, f(·) is typically a transformer encoder or a…”
Exploring wav2vec 2.0 on speaker verification and language identification (2020, methodology): “One kind of method is conducted by the reconstruction loss, such as autoregressive predictive coding (APC) [13], masked predictive coding (MPC) [14] and so on.”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, methodology): “For Mandarin pre-training, we first collected reading type dataset OpenMandarin as described in [18].”
SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering (2019, methodology): “Some more recent works [21, 22, 23, 24, 25, 26] even tried to obtain embeddings for audio signals using approaches very similar to BERT, but primarily extracting acoustic information with very limited semantics.”