Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networks

By Xingcheng Song, Guangsen Wang, Zhiyong Wu, Yiheng Huang, Dan Su, Dong Yu, Helen Mei-Ling Meng. 2019

54 citations recorded by the source corpus.

Self-attention network (SAN) can benefit significantly from the bi-directional representation learning through unsupervised pretraining paradigms such as BERT and XLNet. In this paper, we present an XLNet-like pretraining scheme "Speech-XLNet" for unsupervised acoustic model pretraining to learn speech representations with SAN. The pretrained SAN is finetuned under the hybrid SAN/HMM framework. We conjecture that by shuffling the speech frame orders, the permutation in Speech-XLNet serves as a strong regularizer to encourage the SAN to make inferences by focusing on global structures through its attention weights. In addition, Speech-XLNet also allows the model to explore the bi-directional contexts for effective speech representation learning. Experiments on TIMIT and WSJ demonstrate that Speech-XLNet greatly improves the SAN/HMM performance in terms of both convergence speed and recognition accuracy compared to the one trained from randomly initialized weights. Our best systems achieve a relative improvement of 11.9% and 8.3% on the TIMIT and WSJ tasks respectively. In particular, the best system achieves a phone error rate (PER) of 13.3% on the TIMIT test set, which to our best knowledge, is the lowest PER obtained from a single system.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “…has also been investigated and shown promising results, beneﬁting from richly learned representations (Chung and Glass, 2018; Chuang et al., 2020; Song et al., 2019; Baevski et al., 2020; Wang et al., 2021; Hsu et al., 2021; Chung et al., 2021a), such as wav2vec 2.0 (Baevski et al., 2020) and…”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, background): “for example unidirectional (or autoregressive) reconstruction of magnitude or phase from past frames [8]–[10], [15]–[18], [23], or bidirectional reconstruction of a temporal frame from both past and future slices [12]–[14], [20]–[22], [24]–[27].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “In speech, instead of prediction, we randomly mask temporal slices of acoustic features and attempt to reconstruct them [21, 22, 23, 24, 25, 16].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “and conventionally used for evaluation, while their audio streams are also aggregated to build diversified and large-scale corpora for unsupervised pre-training [109], [123], [218].”
SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering (2019, methodology): “Some more recent works [21, 22, 23, 24, 25, 26] even tried to obtain embeddings for audio signals using approaches very similar to BERT, but primarily extracting acoustic information with very limited semantics.”
Improved Speech Representations with Multi-Target Autoregressive Predictive Coding (2020, background): “…to higher-level representations using only unlabeled speech, has received great attention recently (Baevski et al., 2019, 2020; Liu et al., 2020; Song et al., 2019; Jiang et al., 2019; Schneider et al., 2019; Chorowski et al., 2019; Pascual et al., 2019; Oord et al., 2018; Kamper, 2019; Chen et…”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, methodology): “Other studies [12, 13, 14, 15, 16, 17] were motivated by NLP and applied similar methods on speech tasks.”
A Review on Large Language Models: Architectures, Applications, Taxonomies, Open Issues and Challenges (2024, methodology): “Speech-XLNet [108] is a method for training unsupervised acoustic models to learn speech representations using a Self-Attention Network (SAN) and subsequently fine-tuning it within the hybrid SAN/HMM framework.”
Audio Albert: A Lite Bert for Self-Supervised Learning of Audio Representation (2020, background): “Speech-XLNet [16] is the audio version of XLNet. vq-wav2vec [17] incorporates vector quan-a”