A study of speaker adaptation for DNN-based speech synthesis

By Zhizheng Wu, Pawel Swietojanski, Christophe Veaux, S. Renals, Simon King. 2015

130 citations recorded by the source corpus.

A major advantage of statistical parametric speech synthesis (SPSS) over unit-selection speech synthesis is its adaptability and controllability in changing speaker characteristics and speaking style. Recently, several studies using deep neural networks (DNNs) as acoustic models for SPSS have shown promising results. However, the adaptability of DNNs in SPSS has not been systematically studied. In this paper, we conduct an experimental analysis of speaker adaptation for DNN-based speech synthesis at different levels. In particular, we augment a low-dimensional speaker-specific vector with linguistic features as input to represent speaker identity, perform model adaptation to scale the hidden activation weights, and perform a feature space transformation at the output layer to modify generated acoustic features. We systematically analyse the performance of each individual adaptation technique and that of their combinations. Experimental results confirm the adaptability of the DNN, and listening tests demonstrate that the DNN can achieve significantly better adaptation performance than the hidden Markov model (HMM) baseline in terms of naturalness and speaker similarity.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Frank K. Soong
5. Junichi Yamagishi
6. Sheng Zhao
7. Wei Ping
8. Lei He

Citations in the corpus

• A Survey on Neural Speech Synthesis (2021, background): “, a lot of works in statistic parametric speech synthesis have studied voice adaptation [77, 386, 443, 65, 122], and the recent voice cloning challenge also attracts a lot of participants [388, 118, 331, 45].”
• Neural Voice Cloning with a Few Samples (2018, methodology): “rectly related to speech synthesis. Lastly, speaker-dependent modeling is essential for multi-speaker speech synthesis. Using i-vectors to represent speaker-dependent characteristics is one approach [Wu et al., 2015], however, they have the limitation of being separately trained, with an objective t”
• Deep Voice 2: Multi-Speaker Neural Text-to-Speech (2017, background): “For further context, Wu et al. (2015) empirically studies DNN-based multi-speaker modeling.”
• A Survey of Deep Learning and Its Applications: A New Paradigm to Machine Learning (2019, background): “• Document Analysis & Recognition Wu et al. [78] presented a new approach for the visual and speech recognition.”
• Wasserstein GAN and Waveform Loss-Based Acoustic Model Training for Multi-Speaker Text-to-Speech Synthesis Systems Using a WaveNet Vocoder (2018, methodology): “[19] suggested using i-vectors for modeling multiple speakers and controlling the speaker identity”
• NAUTILUS: A Versatile Voice Cloning System (2020, background): “The speaker representation can be an i-vector [47],”
• Machine Speech Chain with One-shot Speaker Adaptation (2018, background): “[24] performed a systematic speaker adaptation for DNN-based speech synthesis at different levels.”
• Voice Morphing that improves TTS quality using an optimal dynamic frequency warping-and-weighting transform (2016, background): “Recent developments in TTS allow us to significantly improve the quality of the synthesized speech, namely by using an LSTM to predict the acoustic parameters from the linguistic ones [10] and Vocaine as the vocoder [2].”
• Median-based generation of synthetic speech durations using a non-parametric approach (2016, unknown): “A major motivation for this is that such a joint model would allow the simultaneous adaptation [4] and control [5] of rhythmic, melodic and phonetic characteristics in a stable and consistent way.”
• Non-parallel voice conversion using i-vector PLDA: towards unifying speaker verification and transformation (2017, background): “Even though i-vectors have been used for speaker adaptation in deep neural network (DNN) based speech synthesis [10], our perspective to voice conversion is new.”
• Spoken Content and Voice Factorization for Few-Shot Speaker Adaptation (2020, background): “Many researchers have tried to make the acoustic model more interpretable so as to adapt new speaker with small hypothesis space [5–7].”
• Bi-Level Speaker Supervision for One-Shot Speech Synthesis (2020, background): “When the corpus is only tens of minutes, many speaker adaptive solutions have also been proposed to synthe-size speech of target speaker [10–14] and those methods can get effective results.”
• Model architectures to extrapolate emotional expressions in DNN-based text-to-speech (2021, methodology): “Several methods have been proposed to control speakers or emotions, for example, one-hot vector [8, 11], i-vector [7], d-vector [21], x-vector [22].”
• Speaker Representations for Speaker Adaptation in Multiple Speakers' BLSTM-RNN-Based Speech Synthesis (2016, background): “The second one is to control the speaker identity with specially designed hidden layers, such as learning hidden unit contribution (LHUC) [7].”
• Deep Learning for Mandarin-Tibetan Cross-Lingual Speech Synthesis (2022, methodology): “Although DNN-based speaker adaptation [22], [23] has been adopted in various neural networks-based TTS applications [20], [21], the adaptation method has not been applied to the MandarinTibetan cross-lingual speech synthesis.”
• Speaker Adaptation in DNN-Based Speech Synthesis Using d-Vectors (2017, methodology): “In [8], the authors study three different forms of conducting speaker adaptation under a DNN-TTS framework: (1) using i-vectors [14] as speaker-id input feature for the DNN; (2) network manipulation; and (3) conversion of the predicted features using Gaussian mixture models (GMMs).”