Generative adversarial network-based postfilter for statistical parametric speech synthesis

By Takuhiro Kaneko, Hirokazu Kameoka, Nobukatsu Hojo, Yusuke Ijima, Kaoru Hiramatsu, Kunio Kashino. 2016

128 citations recorded by the source corpus.

We propose a postfilter based on a generative adversarial network (GAN) to compensate for the differences between natural speech and speech synthesized by statistical parametric speech synthesis. In particular, we focus on the differences caused by over-smoothing, which makes the sounds muffled. Over-smoothing occurs in the time and frequency directions and is highly correlated in both directions, and conventional methods based on heuristics are too limited to cover all the factors (e.g., global variance was designed only to recover the dynamic range). To solve this problem, we focus on “spectral texture”, i.e., the details of the time-frequency representation, and propose a learning-based postfilter that captures the structures directly from the data. To estimate the true distribution, we utilize a GAN composed of a generator and a discriminator. This optimizes the generator to produce samples imitating the dataset according to the adversarial discriminator. This adversarial process encourages the generator to fit the true data distribution, i.e., to generate realistic spectral texture. Objective evaluation of experimental results shows that the GAN-based postfilter can compensate for detailed spectral structures including modulation spectrum, and subjective evaluation shows that its generated speech is comparable to natural speech.

Subfields: Generative Models, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Generative Models

In Speech & Audio

Citations in the corpus

VQVAE Unsupervised Unit Discovery and Multi-scale Code2Spec Inverter for Zerospeech Challenge 2019 (2019, background): “The discriminator predicts whether the given spectrogram is real data or is generated by the inverter, which generates a realistic spectrogram to deceive the discriminator [14, 15, 16].”
Statistical parametric speech synthesis using generative adversarial networks under a multi-task learning framework (2017, methodology): “In [24], GAN was treated as a post-filter for acoustic models to overcome the over-smoothing problem.”
Adaptive Wavenet Vocoder for Residual Compensation in GAN-Based Voice Conversion (2018, methodology): “GAN has shown to be effective in many fields including speech enhancement [23, 24], speech synthesis [25,26].”
Glow-WaveGAN: Learning Speech Representations from GAN-based Variational Auto-Encoder For High Fidelity Flow-based Speech Synthesis (2021, methodology): “To overcome the above issue, generative adversarial networks are utilized to model the distribution of target speech [13, 14].”
StarGAN-VC: non-parallel many-to-many Voice Conversion Using Star Generative Adversarial Networks (2018, methodology): “While they have been found to be effective for use with image generation, in recent years they have also been employed with notable success for various speech processing tasks [7, 34–38].”
Statistical Parametric Speech Synthesis Incorporating Generative Adversarial Networks (2017, background): “[42] proposed a generative adversarial network-based post-filter for TTS.”
WGANSing: A Multi-Voice Singing Voice Synthesizer Based on the Wasserstein-GAN (2019, methodology): “Other examples of the application of GANs for speech synthesis include [11] and [12], which use GANs as a post-filter for acoustic models to overcome the oversmoothing related to the models used.”
CycleGAN-VC: Non-parallel Voice Conversion Using Cycle-Consistent Adversarial Networks (2018, methodology): “Implementation details: We designed a network based on the recent success in image modeling [22], [48], [49] and speech modeling [8], [29].”
Neural Harmonic-plus-Noise Waveform Model with Trainable Maximum Voice Frequency for Text-to-Speech Synthesis (2019, background): “In text-to-speech (TTS) systems using statistical parametric speech synthesis [1], neural-network (NN)-based models have been introduced to both the front-end text analyzer and the back-end acoustic models [2, 3, 4, 5, 6].”
Conditional Spoken Digit Generation with StyleGAN (2020, methodology): “For the acoustic modeling task, adversarial training has been used as an auxiliary objective alongside conventional regression [8, 9] or as a generative post-filter to add stochasticity to regression-based predictions [10].”
Generative Adversarial Network-Based Glottal Waveform Model for Statistical Parametric Speech Synthesis (2017, methodology): “Results of [24] showed that GANs are capable of producing detailed speech spectra, including also the modulation spectrum, resulting in increased synthesis quality.”
DEEP NEURAL (unknown): “Some examples are statistical parametric speech synthesis [9], learning representations of emotional speech [10], noise reduction in low-dose computed tomography (CT) [11], and retinal image synthesis [12].”
A Comparison of Recent Waveform Generation and Acoustic Modeling Methods for Neural-Network-Based Speech Synthesis (2018, unknown): “Scaling layer in GAN scales each dimension of input features.”