Multi-speaker modeling and speaker adaptation for DNN-based TTS synthesis

By Yuchen Fan, Yao Qian, Frank K. Soong, Lei He. 2015

132 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

A Survey on Neural Speech Synthesis (2021, background): “, a lot of works in statistic parametric speech synthesis have studied voice adaptation [77, 386, 443, 65, 122], and the recent voice cloning challenge also attracts a lot of participants [388, 118, 331, 45].”
Deep Voice 2: Multi-Speaker Neural Text-to-Speech (2017, background): “Similarly, Fan et al. (2015) uses a shared hidden representation among different speakers with speaker-dependent output layers predicting vocoder parameters (e.g., line spectral pairs, aperiodicity parameters etc.).”
SpeechNet: A Universal Modularized Model for Speech Processing Tasks (2021, methodology): “We perform speaker-adaptive TTS (Fan et al., 2015), where the model synthesizes speech conditioned on speaker characteristics.”
Training Multi-Speaker Neural Text-to-Speech Systems using Speaker-Imbalanced Speech Corpora (2019, background): “The multi-speaker model is simple to set up [12, 13] and can generate more stable speech waveforms than those of the speaker-dependent model when the amount of the target speaker’s data is limited [10].”
Linear Networks Based Speaker Adaptation for Speech Synthesis (2018, methodology): “The remainder of this paper is organized as follows: section 2 introduces the adaptation framework in this paper, including 1) the multi-task DNN-BLSTM source acoustic model for speech synthesis baseline, 2) describing the framework of LN based speaker adaptation and 3) introducing LRPD decompositio”
Speaker Adaptation of a Multilingual Acoustic Model for Cross-Language Synthesis (2020, background): “Even though similar techniques have been proposed for DNN-based speech synthesis [15, 17, 16], most works in the literature have focused on monolingual systems that can generate multiple voices [22, 23, 24, 25].”
Wasserstein GAN and Waveform Loss-Based Acoustic Model Training for Multi-Speaker Text-to-Speech Synthesis Systems Using a WaveNet Vocoder (2018, background): “[18] proposed multi-speaker speech synthesis using shared speaker-independent layers as well as a speaker-dependent output layer.”
On the training of DNN-based average voice model for speech synthesis (2016, methodology): “In [15], a so-called multi-task learning was applied to give each speaker a private output layer, and the bottom layers were shared by all the speakers.”
Forward–Backward Decoding Sequence for Regularizing End-to-End TTS (2019, unknown): “[1]–[5].”
From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint (2020, methodology): “html embedding to characterize the expected voice and speaking style in the multispeaker TTS system [10, 11, 12], while speaker adaptation can also be used for speaker transfer TTS modeling [13].”
Model architectures to extrapolate emotional expressions in DNN-based text-to-speech (2021, methodology): “The PM is newly proposed and is motivated by a multi-speaker DNN [6] and the emotion additive model [17], where hidden layers are regarded as a linguistic feature transformation shared by all speakers [23].”
Cross-lingual Multi-speaker Text-to-speech Synthesis for Voice Cloning without Using Parallel Corpus for Unseen Speakers (2019, background): “[10] takes the multi-task learning approach and duplicates the output layer for each of its training speakers so that each speaker is trained with its own speakerdependent output layer while sharing other hidden layers in the model.”
Speaker and language factorization in DNN-based TTS synthesis (2016, methodology): “From the objective results, factorized DNN with 2 linguistic-specific layers, 1 shared layer and 1 speaker-specific layer gets the best performance and is similar to the multi-speaker DNN [9], which builds multi-speaker network for Mandarin and English separately.”
Expressive Speech Synthesis via Modeling Expressions with Variational Autoencoder (2018, background): “Many studies have reported that such labels are helpful for modeling complex audio data [5, 6, 4, 7].”
Deep Learning for Mandarin-Tibetan Cross-Lingual Speech Synthesis (2022, background): “The main network architectures of deep learning include deep neural network (DNN) [7]–[9], long short-term memory (LSTM) and bi-directional long shortterm memory (BLSTM) [10].”
AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN (2020, methodology): “Inspired by [27], modules in the light red dotted rectangle in Figure 1 will be ﬁxed and shared for any new speaker.”
Spoken Content and Voice Factorization for Few-Shot Speaker Adaptation (2020, background): “It has been experimentally proved that the shared hidden layers of acoustic model can benefit synthesized speech of each speaker from the knowledge of others [8,12].”
Deep Encoder-Decoder Models for Unsupervised Learning of Controllable Speech Synthesis (2018, background): “Several publications [4]–[7] have meanwhile highlighted the potential beneﬁts of acoustic variation (at least when annotated), for instance [7] presenting multi-speaker synthesisers that are more accurate than could be expected from training on any single speaker in the database alone and…”