Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data

By Wei-Ning Hsu, Yu Zhang, James R. Glass. 2017

376 citations recorded by the source corpus.

We present a factorized hierarchical variational autoencoder, which learns disentangled and interpretable representations from sequential data without supervision. Specifically, we exploit the multi-scale nature of information in sequential data by formulating it explicitly within a factorized hierarchical graphical model that imposes sequence-dependent priors and sequence-independent priors to different sets of latent variables. The model is evaluated on two speech corpora to demonstrate, qualitatively, its ability to transform speakers or linguistic content by manipulating different sets of latent variables; and quantitatively, its ability to outperform an i-vector baseline for speaker verification and reduce the word error rate by as much as 35% in mismatched train/test scenarios for automatic speech recognition tasks.

Subfields: Artificial Intelligence, Generative Models, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Generative Models

In Representation & Self-Supervised Learning

Citations in the corpus

A Sober Look at the Unsupervised Learning of Disentangled Representations and their Evaluation (2020, background): “One can also exploit known relations between factors in different samples (Karaletsos et al., 2015; Goroshin et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton and Birodkar, 2017; Hsu et al., 2017; Yingzhen and Mandt, 2018b; Locatello et al., 2018).”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, background): “Both signal processing-based [11] and learned [12] features have been explored.”
Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, background): “Other work considers a variety of downstream supervised tasks, and some focuses explicitly on learning representations that generalize across tasks or across very different domains [6, 7, 18, 19].”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “…successfully used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018; Chung et al., 2019; Pascual et…”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “[66] explore a hierarchical latent space which separates sequence-dependent variables from those which are sequence-independent ones.”
Weakly-Supervised Disentanglement Without Compromises (2020, background): “For example, a number of works rely on known relations between the factors of variation (Karaletsos et al., 2015; Whitney et al., 2016; Fraccaro et al., 2017; Denton & Birodkar, 2017; Hsu et al., 2017; Yingzhen & Mandt, 2018; Locatello et al., 2018) and disentangling motion and pose from content (Hs”
On the Fairness of Disentangled Representations (2019, methodology): “Weaker forms of supervision like relational information or additional assumptions on the effect of the factors of variation were also studied [39, 16, 47, 31, 91, 26, 19, 41, 93, 62, 53, 81, 8] and applied in the sequential data and reinforcement learning settings [88, 85, 57, 69, 37, 38].”
Are Disentangled Representations Helpful for Abstract Visual Reasoning? (2019, neutral): “Due to the initial poor performance of purely unsupervised methods, the ﬁeld initially focused on semi-supervised [60, 11, 55, 56, 43, 45] and weakly supervised approaches [31, 12, 40, 21, 75, 20, 15, 35, 77, 52, 46, 62, 8].”
Disentangling by Factorising (2018, background): “…or explicit knowledge about the true underlying factors of the data have excelled at disentangling (Kulkarni et al., 2015; Kingma et al., 2014; Reed et al., 2014; Siddharth et al., 2017; Hinton et al., 2011; Mathieu et al., 2016; Goroshin et al., 2015; Hsu et al., 2017; Denton & Birodkar, 2017).”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “The research line of generative learning can be traced back to the auto-encoding model, which reconstructs the whole speech from latent variables, either continuous [25]– [27] or discrete [28].”
Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, background): “Various constraints can be imposed on the encoded space, such as temporal smoothness [10], discreteness [11], and hierarchy [12].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Various constraints can be imposed on the encoded space, such as temporal smoothness (Ebbers et al., 2017; Glarner et al., 2018; Khurana et al., 2019, 2020), discrete-ness (Ondel et al., 2016; van den Oord et al., 2017), and presence of hierarchy (Lee and Glass, 2012; Hsu et al., 2017b).”
Hierarchical Generative Modeling for Controllable Speech Synthesis (2018, methodology): “Experimental results verify the effectiveness, and similar design is used in Hsu et al. (2017a).”
Recent Advances in Autoencoder-Based Representation Learning (2018, background): “Hsu et al. [73] develop a hierarchical VAE model to learn interpretable representations of speech recordings.”
AudioGen: Textually Guided Audio Generation (2022, background): “Various constraints can be imposed on the encoded space, such as temporal smoothness (Ebbers et al., 2017), discreteness (van den Oord et al., 2017b), and hierarchy (Hsu et al., 2017).”