A Neural Vocoder With Hierarchical Generation of Amplitude and Phase Spectra for Statistical Parametric Speech Synthesis

By Yang Ai, Zhenhua Ling. 2019

35 citations recorded by the source corpus.

This article presents a neural vocoder named HiNet which reconstructs speech waveforms from acoustic features by predicting amplitude and phase spectra hierarchically. Different from existing neural vocoders such as WaveNet, SampleRNN and WaveRNN which directly generate waveform samples using single neural networks, the HiNet vocoder is composed of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP). The ASP is a simple DNN model which predicts log amplitude spectra (LAS) from acoustic features. The predicted LAS are sent into the PSP for phase recovery. Considering the issue of phase warping and the difficulty of phase modeling, the PSP is constructed by concatenating a neural source-filter (NSF) waveform generator with a phase extractor. We also introduce generative adversarial networks (GANs) into both ASP and PSP. Finally, the outputs of ASP and PSP are combined to reconstruct speech waveforms by short-time Fourier synthesis. Since there are no autoregressive structures in both predictors, the HiNet vocoder can generate speech waveforms with high efficiency. Objective and subjective experimental results show that our proposed HiNet vocoder achieves better naturalness of reconstructed speech than the conventional STRAIGHT vocoder, a 16-bit WaveNet vocoder using open source implementation and an NSF vocoder with similar complexity to the PSP and obtains similar performance with a 16-bit WaveRNN vocoder. We also find that the performance of HiNet is insensitive to the complexity of the neural waveform generator in PSP to some extend. After simplifying its model structure, the time consumed for generating 1 s waveforms of 16 kHz speech using a GPU can be further reduced from 0.34 s to 0.19 s without significant quality degradation.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Sheng Zhao
6. Zhenhua Ling
7. Yang Ai
8. Yi Ren

In Computer Vision and Pattern Recognition

1. Tie-Yan Liu
2. Tao Qin
3. Xu Tan
4. Frank K. Soong
5. Yang Ai
6. Zhenhua Ling
7. Yingce Xia
8. Di He

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Zhenhua Ling
6. Yang Ai
7. Sheng Zhao
8. Yi Ren

In Signal Processing

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Yang Ai
6. Zhenhua Ling
7. Sheng Zhao
8. Yi Ren

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, methodology): “Some works [50, 3, 62, 322, 210, 212] investigate different model structures such as CRF [176], RNN [111], and self-attention [362] for prosody prediction in Chinese.”
• Knowledge-and-Data-Driven Amplitude Spectrum Prediction for Hierarchical Neural Vocoders (2020, methodology): “In our previous work [27], we proposed a neural vocoder named HiNet, which consists of an amplitude spectrum predictor (ASP) and a phase spectrum predictor (PSP).”