Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion

By Andy T. Liu, Po-Chun Hsu, Hung-yi Lee. 2019

30 citations recorded by the source corpus.

We present an unsupervised end-to-end training scheme where we discover discrete subword units from speech without using any labels. The discrete subword units are learned under an ASR-TTS autoencoder reconstruction setting, where an ASR-Encoder is trained to discover a set of common linguistic units given a variety of speakers, and a TTS-Decoder trained to project the discovered units back to the designated speech. We propose a discrete encoding method, Multilabel-Binary Vectors (MBV), to make the ASR-TTS autoencoder differentiable. We found that the proposed encoding method offers automatic extraction of speech content from speaker style, and is sufficient to cover full linguistic content in a given language. Therefore, the TTS-Decoder can synthesize speech with the same content as the input of ASR-Encoder but with different speaker characteristics, which achieves voice conversion (VC). We further improve the quality of VC using adversarial training, where we train a TTS-Patcher that augments the output of TTS-Decoder. Objective and subjective evaluations show that the proposed approach offers strong VC results as it eliminates speaker identity while preserving content within speech. In the ZeroSpeech 2019 Challenge, we achieved outstanding performance in terms of low bitrate.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “As a result, the embedding dimension of each code and the number of codes need to be carefully tuned (Liu et al., 2019).”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “…low bitrate speech representations in an unsupervised manner, has attracted attention from both the machine learning and the speech communities (Liu et al., 2019a; Feng et al., 2019; Nayak et al., 2019; Tjan-dra et al., 2019; Schneider et al., 2019; Baevski et al., 2020a; Chen and Hain, 2020;…”
The Zero Resource Speech Challenge 2019: TTS without T (2019, background): “FE [27] no no Ossian LI [28] yes yes Inversion”
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders (2019, background): “Unsupervised speech representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10] is effective in extracting high-level properties from speech.”
How Far Are We from Robust Voice Conversion: A Survey (2020, background): “Zeroshot approaches [8, 9, 10, 11, 12] are then considered, where the models can perform VC among any speakers without fine-tuning.”
Speech Representation Disentanglement with Adversarial Mutual Information Learning for One-shot Voice Conversion (2022, methodology): “The unsupervised end-to-end automatic speech recognition and text-to-speech (ASR-TTS) autoencoder framework [10] use multilabel-binary vectors to represent the content of speech to dientangle content from speaker.”