Learning Latent Representations for Speech Generation and Transformation

By Wei-Ning Hsu, Yu Zhang, James R. Glass. 2017

146 citations recorded by the source corpus.

An ability to model a generative process and learn a latent representation for speech in an unsupervised fashion will be crucial to process vast quantities of unlabelled speech data. Recently, deep probabilistic generative models such as Variational Autoencoders (VAEs) have achieved tremendous success in modeling natural images. In this paper, we apply a convolutional VAE to model the generative process of natural speech. We derive latent space arithmetic operations to disentangle learned latent representations. We demonstrate the capability of our model to modify the phonetic content or the speaker identity for speech segments using the derived operations, without the need for parallel supervisory data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, methodology): “distributions with more powerful neural network models [29]– [33].”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, methodology): “…successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018; Chung et al., 2019; Pascual et…”
An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “There also exists some work on unsupervised speech representation learning [11, 12, 13, 14, 15, 16].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Reconstruction is often implemented in the form of autoencoding (Hsu et al., 2017a), where speech is first encoded into a low-dimensional space, and then decoded back to speech.”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017, background): “There is yet another line of research analyzing the latent variables with labeled data after the model is trained [33, 15].”
Taming VAEs (2018, background): “Deep variational auto-encoders (VAEs) have made substantial progress in the last few years on modelling complex data such as images [1, 2, 3, 4, 5, 6, 7], speech synthesis [8, 9], molecular discovery [10, 11, 12], robotics [13, 14, 15] and 3d-scene understanding [16, 17, 18].”
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition (2018, methodology): “In [18], a convolutional variational autoencoder (VAE) is proposed to model such process; by assuming the prior to be a diagonal Gaussian, it is shown that the VAE automatically learns to model independent attributes regarding generation, such as the speaker identity and the linguistic content, usin”
Variational Autoencoder with Implicit Optimal Priors (2018, methodology): “Since the VAE can capture the high-dimensional complicated data distributions, it is widely applied to various data, such as images (Gulrajani et al. 2016), videos (Gregor et al. 2015), and audio and speech (Hsu, Zhang, and Glass 2017; van den Oord, Vinyals, and kavukcuoglu 2017).”
Learning modality-invariant representations for speech and images (2017, methodology): “[11] designed a convolutional variational autoencoder (CVAE) for log melfilterbanks of speech drawn from the TIMIT dataset.”
Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder (2018, background): “, [14]) have shown VAEs can model global characteristics of speech such as speaker individualities, but to our knowledge no study has yet suggested using VAEs for SS or UESS.”
Scalable Factorized Hierarchical Variational Autoencoder Training (2018, background): “A learned latent representation from speech can be useful for many downstream applications, such speaker verification [1], automatic speech recognition [2], and linguistic unit discovery [3, 4].”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, background): “The earliest line of work relies on generative modeling with latent variables [24, 25, 8, 26, 27, 28].”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, methodology): “Some LVMs do not use autoregressive observation models (Ebbers et al. 2017; Glarner et al. 2018; Hsu, Zhang, and Glass 2017b,a; Khurana et al. 2019, 2020).”
Generative timbre spaces: regularizing variational auto-encoders with perceptual metrics (2018, background): “Their potential for audio applications have been only scarcely investigated yet and mostly in topics related to speech processing such as blind source separation [18] and speech transformation [19].”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “…2020a), while others have implemented autoen-coder models with various constraints applied to the latent representations, such as discrete-ness (van den Oord et al., 2017; Eloff et al., 2019; Chorowski et al., 2019) or disentanglement (Hsu et al., 2017c; Hsu and Glass, 2018b; Khurana et al., 2019).”
Cyclic Spectral Modeling for Unsupervised Unit Discovery into Voice Conversion with Excitation and Waveform Modeling (2020, methodology): “Secondly, we measured the rootmean-square error (RMSE) and cosine-similarity [36] between the VQ latent features estimated from the source utterances and that from the target utterances.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “VAEs have been applied to speech in numerous works [70], [71], [72], [73], [74].”