An Unsupervised Autoregressive Model for Speech Representation Learning

By Yu-An Chung, Wei-Ning Hsu, Hao Tang, James R. Glass. 2019

432 citations recorded by the source corpus.

This paper proposes a novel unsupervised autoregressive neural model for learning generic speech representations. In contrast to other speech representation learning methods that aim to remove noise or speaker variabilities, ours is designed to preserve information for a wide range of downstream tasks. In addition, the proposed model does not require any phonetic or word boundary labels, allowing the model to benefit from large quantities of unlabeled data. Speech representations learned by our model significantly improve performance on both phone classification and speaker verification over the surface features and other supervised and unsupervised approaches. Further analysis shows that different levels of speech information are captured by our model at different layers. In particular, the lower layers tend to be more discriminative for speakers, while the upper layers provide more phonetic content.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, background): “Other related work includes learning representations from auto-encoding the input data [52, 11] or directly predicting future timesteps [8].”
Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, neutral): “The typical generative method is Autoregressive Predictive Coding (APC) [8, 9], where the model is similar to the autoen-coder architectures except that the network is trained to predict features for future time steps.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “Examples of pretext tasks for self-supervised speech representation learning include distinguishing near-by features from temporally distant ones [1]–[3], next-step prediction of audio features [4], masked prediction of audio features given unmasked context [5], [6].”
Unsupervised Speech Recognition (2021, background): “…years due to advances in model architectures (Dong et al., 2018; Synnaeve et al., 2020; Gulati et al., 2020), semisupervised learning (Xu et al., 2020b; Park et al., 2020) and self-supervised learning (van den Oord et al., 2018; Chung and Glass, 2018; Chung et al., 2019b; Baevski et al., 2020c).”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, background): “Self-supervised learning of speech representations has received a lot attention [1, 2, 3, 4, 5] and has been demonstrated to work well both in low- and high-resource labeled data settings for automatic speech recognition (ASR) [6].”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Another line of work is pre-training representations on unlabeled speech followed by fine-tuning on labeled data [16, 17, 18, 19, 20, 21, 22, 23, 24].”
Improved Noisy Student Training for Automatic Speech Recognition (2020, background): “In this approach, a separate task/loss to (pre-)train the acoustic model (or a part thereof) is introduced to help the model learn a good representation of the input, e.g., [24, 25, 26, 27, 28, 29, 30].”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, background): “This work uses a variety of training objectives, including autoencoderbased [15] and language model-like [7].”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “Deep neural network models have been successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018; ”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, neutral): “Apart from CPC, [10] proposed a different pre-training model called Autore-gressive Predictive Coding (APC) and also got comparable results on phoneme classiﬁcation and speaker veriﬁcation.”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, background): “A popular approach to discover discrete units is via autoencoding (Tjandra et al., 2019; Eloff et al., 2019; Chorowski et al., 2019) sometimes coupled with an autoregressive model (Chung et al., 2019).”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Examples of information include spectrograms (Chung et al., 2019; Wang et al., 2020; Chi et al., 2020; Liu et al., 2020; Chung and Glass, 2020; Liu et al., 2020; Ling et al., 2020; Ling and Liu, 2020), cluster indices (Baevski and Mohamed, 2020; Hsu et al.”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Signiﬁcant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are ﬁne-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, background): “In speech processing, most work in this area has focused on monolingual unsupervised representation learning [18, 19, 20, 21, 22, 23, 24, 25].”