A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition

By Dongwei Jiang, Wubo Li, Ruixiong Zhang, Miao Cao, Ne Luo, Yang Han, Wei Zou, Xiangang Li. 2020

32 citations recorded by the source corpus.

The construction of an effective good speech recognition system typically requires large amounts of transcribed data, which is expensive to collect. To overcome this problem, many unsupervised pretraining methods have been proposed. Among these methods, Masked Predictive Coding achieved significant improvements on various speech recognition datasets with BERT-like Masked Reconstruction loss and transformer backbone. However, many aspects of MPC have yet to be fully investigated. In this paper, we conduct a further study on MPC and focus on three important aspects: the effect of pretraining data speaking style, its extension on streaming model, and strategies for better transferring learned knowledge from pretraining stage to downstream tasks. The experimental results demonstrated that pretraining data with a matching speaking style is more useful on downstream recognition tasks. A unified training objective with APC and MPC provided an 8.46% relative error reduction on the streaming model trained on HKUST. Additionally, the combination of target data adaption and layerwise discriminative training facilitated the knowledge transfer of MPC, which realized 3.99% relative error reduction on AISHELL over a strong baseline.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “, 2020), APC (Chung and Glass, 2020), MPC (Jiang et al., 2021), and Hubert (Hsu et al.”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “In [25], a simpler masking policy is employed, where input features are divided into chunks of four frames, and masking on chunks are applied with a probability of 15%.”
BENDR: Using Transformers and a Contrastive Self-Supervised Learning Task to Learn From Massive Amounts of EEG Data (2021, methodology): “Prior work in self-supervised speech recognition has begun to synthesize parts of CPC and MLM to produce methodologies for self-learning with raw waveforms (van den Oord et al., 2018; Baevski and Mohamed, 2020; Baevski et al., 2020; Chung et al., 2020; Jiang et al., 2020).”
Didispeech: A Large Scale Mandarin Speech Corpus (2020, methodology): “However, the LibriTTS is a subset of the LibriSpeech [24] database which is designed for ASR tasks.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “For the purpose of pre-training and evaluating Mandarin speech representations, the authors of [101], [104] also compiled Open Mandarin, an open-source Mandarin dataset of 1.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “…autoregressive language models (Mikolov et al. 2010), contextualized speech representations can be learned by predicting future values of a simple representation (van den Oord, Li, and Vinyals 2018; Chung et al. 2019; Schneider et al. 2019; Chung and Glass 2020a; Jiang et al. 2021) (PRD, table 1).”