BOFFIN TTS: Few-Shot Speaker Adaptation by Bayesian Optimization
By Henry B. Moss, Vatsal Aggarwal, Nishant Prateek, Javier I. González, Roberto Barra-Chicote. 2020
59 citations recorded by the source corpus.
We present BOFFIN TTS (Bayesian Optimization For FIne-tuning Neural Text To Speech), a novel approach for few-shot speaker adaptation. Here, the task is to fine-tune a pre-trained TTS model to mimic a new speaker using a small corpus of target utterances. We demonstrate that there does not exist a one-size-fits-all adaptation strategy, with convincing synthesis requiring a corpus-specific configuration of the hyper-parameters that control fine-tuning. By using Bayesian optimization to efficiently optimize these hyper-parameter values for a target speaker, we are able to perform adaptation with an average 30% improvement in speaker similarity over standard techniques. Results indicate, across multiple corpora, that BOFFIN TTS can learn to synthesize new speakers using less than ten minutes of audio, achieving the same naturalness as produced for the speakers used to train the base model.
Subfields: Artificial Intelligence, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Signal Processing
Citations in the corpus
- A Survey on Neural Speech Synthesis (2021, methodology): “Some works [43, 9, 173, 234, 439, 48, 45, 39, 230] conduct few-shot adaptation that only uses few paired text and speech data, varying from several minutes to several seconds.”
- AdaSpeech: Adaptive Text to Speech for Custom Voice (2021, background): “They fine-tune the whole model (Chen et al., 2018; Kons et al., 2019) or decoder part (Moss et al., 2020; Zhang et al., 2020), achieving good quality but causing too many adaptation parameters.”
- Meta-StyleSpeech : Multi-Speaker Adaptive Text-to-Speech Generation (2021, background): “A popular approach is to train the model on a large multi-speakers dataset and then fine-tune the whole model (Chen et al., 2019; Arik et al., 2018) or only parts of the model (Moss et al., 2020; Zhang et al., 2020; Chen et al., 2021).”
- AdaDurIAN: Few-shot Adaptation for Neural Text-to-Speech with DurIAN (2020, background): “…embedding and combine it with linguistic feature as inputs to a TTS model [17, 18], which may require a very strong speaker encoder network trained by thousands of speakers [19]; the second is to fine-tune the entire multi-speaker network to select a optimal single-speaker model [20, 21, 22].”
- Data Efficient Voice Cloning from Noisy Samples with Domain Adversarial Training (2020, methodology): “The study in [9] demonstrates that the training strategy cannot be fixed for adaptation of different speakers and presents a Bayesian optimization method for fine-tuning the TTS model.”
- Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “[142] proposed a fine-tuning method to select different model hyperparameters for different speakers, achieving the goal of synthesizing the voice of a specific speaker with only a small number of speech samples, in which the selection of hyperparameters adopts the Bayesian optimization method [184]”
- Unsupervised Learning For Sequence-to-sequence Text-to-speech For Low-resource Languages (2020, background): “Some proposed to fine-tune all or parts of the pre-trained model using a small amount of data from target speakers [6,7].”
- Using IPA-Based Tacotron for Data Efficient Cross-Lingual Speaker Adaptation and Pronunciation Enhancement (2020, background): “Another way of speaker adaptation is based on fine-tuning the weights of an already trained model [7, 8].”