Unsupervised acoustic unit discovery for speech synthesis using discrete latent-variable neural networks

By Ryan Eloff, A. Nortje, Benjamin van Niekerk, Avashna Govender, Leanne Nortje, Arnu Pretorius, Elan Van Biljon, E. V. D. Westhuizen, Lisa van Staden, Herman Kamper. 2019

59 citations recorded by the source corpus.

For our submission to the ZeroSpeech 2019 challenge, we apply discrete latent-variable neural networks to unlabelled speech and use the discovered units for speech synthesis. Unsupervised discrete subword modelling could be useful for studies of phonetic category learning in infants or in low-resource speech technology requiring symbolic input. We use an autoencoder (AE) architecture with intermediate discretisation. We decouple acoustic unit discovery from speaker modelling by conditioning the AE's decoder on the training speaker identity. At test time, unit discovery is performed on speech from an unseen speaker, followed by unit decoding conditioned on a known target speaker to obtain reconstructed filterbanks. This output is fed to a neural vocoder to synthesise speech in the target speaker's voice. For discretisation, categorical variational autoencoders (CatVAEs), vector-quantised VAEs (VQ-VAEs) and straight-through estimation are compared at different compression levels on two languages. Our final model uses convolutional encoding, VQ-VAE discretisation, deconvolutional decoding and an FFTNet vocoder. We show that decoupled speaker conditioning intrinsically improves discrete acoustic representations, yielding competitive synthesis quality compared to the challenge baseline.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, neutral): “Other related work includes learning representations from auto-encoding the input data [52, 11] or directly predicting future timesteps [8].”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “…et al., 2019a; Merkx et al., 2019; Chrupała et al., 2017; Scharenborg et al., 2018; Kamper et al., 2019b;a; Surı́s et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a), application to multilingual settings (Harwath et al., 2018a; Kamper & Roth, 2017; Azuh et al., 2019; Havard et al., 2019a),…”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, methodology): “A popular approach to discover discrete units is via autoencoding (Tjandra et al., 2019; Eloff et al., 2019; Chorowski et al., 2019) sometimes coupled with an autoregressive model (Chung et al., 2019).”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “In [18] the authors proposed a VQ-VAE model followed by an FFTNet vocoder model [19].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Eloff et al. (2019) suggested a VQ-VAE model followed by a FFTNet vocoder model (Jin et al., 2018).”
The Zero Resource Speech Challenge 2019: TTS without T (2019, background): “EL [29] no yes FFTnet NA [30] no no Ossian”
Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge (2020, methodology): “The first is the VQ-VAE-based system we submitted to the previous challenge [21], referred to here as VQ-VAE(spec).”
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, methodology): “…task for unsupervised pre-training. vq-wav2vec (Baevski et al., 2020a) proposes to add a vector quantizer similar to VQ-VAE (van den Oord et al., 2018), using Gumbel softmax (Jang et al., 2016) or online k-means clustering to quantize the dense speech representations (Eloff et al., 2019).”
Cyclic Spectral Modeling for Unsupervised Unit Discovery into Voice Conversion with Excitation and Waveform Modeling (2020, methodology): “A lot of work have been proposed for the unsupervised discovery of subword units [3, 6, 8, 11, 12, 13].”
Transformer VQ-VAE for Unsupervised Unit Discovery and Speech Synthesis: ZeroSpeech 2020 Challenge (2020, methodology): “Following our success in a previous challenge [2] in ZeroSpeech 2019 and several prior publications on quantization-based approaches [3, 4, 5], we used a vector quantized variational autoencoder (VQVAE) as the primary model and a codebook inverter as the secondary model.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “As described above, AEs consist of an encoder and decoder; the pretext task is to reconstruct the given input.”
Unsupervised Acoustic Unit Representation Learning for Voice Conversion using WaveNet Auto-encoders (2020, background): “However, it is notable that VQVAE based systems [24, 25] suffered from poor quality of the reconstructed speech.”