Scalable Factorized Hierarchical Variational Autoencoder Training

By Wei-Ning Hsu, James R. Glass. 2018

26 citations recorded by the source corpus.

Deep generative models have achieved great success in unsupervised learning with the ability to capture complex nonlinear relationships between latent generating factors and observations. Among them, a factorized hierarchical variational autoencoder (FHVAE) is a variational inference-based model that formulates a hierarchical generative process for sequential data. Specifically, an FHVAE model can learn disentangled and interpretable representations, which have been proven useful for numerous speech applications, such as speaker verification, robust speech recognition, and voice conversion. However, as we will elaborate in this paper, the training algorithm proposed in the original paper is not scalable to datasets of thousands of hours, which makes this model less applicable on a larger scale. After identifying limitations in terms of runtime, memory, and hyperparameter optimization, we propose a hierarchical sampling training algorithm to address all three issues. Our proposed method is evaluated comprehensively on a wide variety of datasets, ranging from 3 to 1,000 hours and involving different types of generating factors, such as recording conditions and noise types. In addition, we also present a new visualization method for qualitatively evaluating the performance with respect to the interpretability and disentanglement. Models trained with our proposed algorithm demonstrate the desired characteristics on all the datasets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Generative Models, Probabilistic & Bayesian Methods, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Generative Models

In Probabilistic & Bayesian Methods

In Signal Processing

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “…used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018; Chung et al., 2019; Pascual et al., 2019).”
Transfer Learning from Audio-Visual Grounding to Speech Recognition (2019, methodology): “A discriminative weight of α = 10 is applied for all models, and the scalable training algorithm proposed in [51] is used for training on PlacesAudCap dataset with a sequence batch size K = 5000, because the original algorithm cannot handle large-scale datasets.”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “Several papers have investigated masked prediction [8, 86, 7, 103], while others have implemented autoencoder models with various constraints applied to the latent representations, such as discreteness [98, 25, 11] or disentanglement [50, 48, 65, 60].”
A Factorial Deep Markov Model for Unsupervised Disentangled Representation Learning from Speech (2019, background): “al [6] propose a training methodology based on hierarchical sampling of the training sequences that makes training an FHVAE possible on large datasets.”