Multi-SpectroGAN: High-Diversity and High-Fidelity Spectrogram Generation with Adversarial Style Combination for Speech Synthesis

By Sang-Hoon Lee, Hyun-Wook Yoon, Hyeong-Rae Noh, Ji-Hoon Kim, Seong-Whan Lee. 2020

60 citations recorded by the source corpus.

While generative adversarial networks (GANs) based neural text-to-speech (TTS) systems have shown significant improvement in neural speech synthesis, there is no TTS system to learn to synthesize speech from text sequences with only adversarial feedback. Because adversarial feedback alone is not sufficient to train the generator, current models still require the reconstruction loss compared with the ground-truth and the generated mel-spectrogram directly. In this paper, we present Multi-SpectroGAN (MSG), which can train the multi-speaker model with only the adversarial feedback by conditioning a self-supervised hidden representation of the generator to a conditional discriminator. This leads to better guidance for generator training. Moreover, we also propose adversarial style combination (ASC) for better generalization in the unseen speaking style and transcript, which can learn latent representations of the combined style embedding from multiple mel-spectrograms. Trained with ASC and feature matching, the MSG synthesizes a high-diversity mel-spectrogram by controlling and mixing the individual speaking styles (e.g., duration, pitch, and energy). The result shows that the MSG synthesizes a high-fidelity mel-spectrogram, which has almost the same naturalness MOS score as the ground-truth mel-spectrogram.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Yi Ren
5. Zhou Zhao
6. Frank K. Soong
7. Sheng Zhao
8. Dong Yu

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Tacotron [72] AM ph FF−→melS AR−→wav arXiv20 2020.10 DeviceTTS [123] AM ph AR−→Ceps→wav arXiv20 2020.10 Wave-Tacotron [379] E2E ch/ph AR−→wav ICASSP21 2020.11 DenoiSpeech [427] AM ph FF−→melS FF−→wav ICASSP21 2020.12 EfficientTTS [229] AM ch FF−→melS FF−→wav ICML21 2020.12 EfficientTTS-Wav [229] E2E”
• Diffsound: Discrete Diffusion Model for Text-to-Sound Generation (2022, background): “In the past few years, Generative Adversarial Networks (GANs) have shown promising results in image generation [16], [17], [18], speech synthesis [6], [19], [20] and music genera-”
• DiffProsody: Diffusion-Based Latent Prosody Generation for Expressive Speech Synthesis With Prosody Conditional Adversarial Training (2023, methodology): “We set multiple PCDs to receive different input sizes, such as [49]–[51].”
• HierSpeech++: Bridging the Gap Between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-Shot Speech Synthesis (2023, unknown): “…and voice similarity MOS (sMOS) with a CI of 95%; and three objective metrics for naturalness: UTMOS [82], character error rate (CER), and word error rate (WER); two objective metrics for similarity: automatic speaker verification equal error rate (EER) and speaker encoder cosine similarity (SECS).”
• Prosospeech: Enhancing Prosody with Quantized Vector Pre-Training in Text-To-Speech (2022, background): “Thanks to the advance of non-autoregressive models [4, 5] and powerful generative models [6, 7, 8, 9], the audio quality and inference speed of modern TTS has been greatly improved.”
• Fre-GAN: Adversarial Frequency-consistent Audio Synthesis (2021, methodology): “While they can produce natural waveform in real-time because of their ability to convert noise sequence into raw waveform in parallel, they require a heavy computation due to the complex architecture [8].”