Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning

By Alexander H. Liu, Tao Tu, Hung-yi Lee, Lin-Shan Lee. 2019

53 citations recorded by the source corpus.

In this paper we propose a Sequential Representation Quantization AutoEncoder (SeqRQ-AE) to learn from primarily unpaired audio data and produce sequences of representations very close to phoneme sequences of speech utterances. This is achieved by proper temporal segmentation to make the representations phoneme-synchronized, and proper phonetic clustering to have total number of distinct representations close to the number of phonemes. Mapping between the distinct representations and phonemes is learned from a small amount of annotated paired data. Preliminary experiments on LJSpeech demonstrated the learned representations for vowels have relative locations in latent space in good parallel to that shown in the IPA vowel chart defined by linguistics experts. With less than 20 minutes of annotated speech, our method outperformed existing methods on phoneme recognition and is able to synthesize intelligible speech that beats our baseline model.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, positive): “As part of training, we learn discrete speech units [53, 32, 7, 18] via a gumbel softmax [24, 5] to represent the latent representations in the contrastive task (Figure 1) which we ﬁ nd to be more effective than non-quantized targets.”
Unsupervised Speech Recognition (2021, background): “Another line of work explored quantization of the continuous speech data to identify speech units in order to learn representations (Baevski et al., 2020b,a; Liu et al., 2019; van Niekerk et al., 2020; Baevski et al., 2020c).”
A Survey on Neural Speech Synthesis (2021, methodology): “In this way, the quantized discrete tokens and the speech can be regarded as pseudo paired data to pre-train a TTS model, which is then fine-tuned on few truly paired text and speech data [197, 352, 429].”
Speak, Read and Prompt: High-Fidelity Text-to-Speech with Minimal Supervision (2023, methodology): “Liu et al. (2020) combine a sequential au-toencoder with vector quantization and temporal segmentation mechanisms to learn a phoneme-like discrete speech representation, along with a seq2seq model that maps these representations to phonemes.”
UWSpeech: Speech to Speech Translation for Unwritten Languages (2020, background): “Among these works, vector quantized variational autoencoder (VQ-VAE) [3, 7, 9, 22, 34–36] has been widely adopted and shown advantages over other methods.”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “Orthogonal to the contrastive-/reconstructive-loss based speech representation learning, vector-quantized speech representations have been proposed [17, 26, 18, 15, 27].”
Vector-Quantized Autoregressive Predictive Coding (2020, background): “Recent studies on VQ for representation learning, mostly motivated by the discrete nature of phonetic units, attempt to show that enforcing the quantization leads to a better representation for acoustic unit discovery [16, 17] and automatic speech recognition [7, 18].”
Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation (2020, methodology): “As for text and audio preprocessing, we follow the prior work [21].”
Multilingual Byte2Speech Text-To-Speech Models Are Few-shot Spoken Language Learners (2021, methodology): “…et al., 2020b), or leveraging other corpora like ﬁne-tuning autoencoder with discrete latents trained on unpaired target speeches (Zhang & Lin, 2020; Liu et al., 2020a), or dual learning between recognition and synthesis, along with denoising autoencoding (Ren et al., 2019), and then added by…”
LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition (2020, background): “• There are also some related works focusing on low-resource TTS and ASR, such as Speech Chain [39], Almost Unsup [31], and SeqRQ-AE [23].”