FedSpeech: Federated Text-to-Speech with Continual Learning

By Ziyue Jiang, Yi Ren, Ming Lei, Zhou Zhao. 2021

31 citations recorded by the source corpus.

Federated learning enables collaborative training of machine learning models under strict privacy restrictions and federated text-to-speech aims to synthesize natural speech of multiple users with a few audio training samples stored in their devices locally. However, federated text-to-speech faces several challenges: very few training samples from each speaker are available, training samples are all stored in local device of each user, and global model is vulnerable to various attacks. In this paper, we propose a novel federated learning architecture based on continual learning approaches to overcome the difficulties above. Specifically, 1) we use gradual pruning masks to isolate parameters for preserving speakers' tones; 2) we apply selective masks for effectively reusing knowledge from tasks; 3) a private speaker embedding is introduced to keep users' privacy. Experiments on a reduced VCTK dataset demonstrate the effectiveness of FedSpeech: it nearly matches multi-task training in terms of multi-speaker speech quality; moreover, it sufficiently retains the speakers' tones and even outperforms the multi-task training in the speaker similarity experiment.

Subfields: Artificial Intelligence, Speech & Audio, Federated & Distributed Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Rongjie Huang
2. Jinglin Liu
3. Zhou Zhao
4. Yi Ren
5. Hang Su
6. Chenye Cui
7. Jun Zhu
8. Liyuan Wang

In Speech & Audio

1. Jinglin Liu
2. Rongjie Huang
3. Zhou Zhao
4. Yi Ren
5. Chenye Cui
6. Jun Zhu
7. Xingxing Zhang
8. Xu Tan

In Federated & Distributed Learning

1. Weiming Zhuang
2. Chen Chen
3. Jie M. Zhang
4. LingJuan Lv

Citations in the corpus

• A Comprehensive Survey of Continual Learning: Theory, Method and Application (2023, background): “[188] adopts a set of gradual pruning masks for parameter allocation and a set of selective masks to reuse old knowledge.”
• TARGET: Federated Class-Continual Learning via Exemplar-Free Distillation (2023, background): “To address these issues, recent research [36, 11, 3, 19] has introduced the concept of Continual Learning (CL) [61, 42, 51, 49, 48] within the FL framework.”
• GenerSpeech: Towards Style Transfer for Generalizable Out-Of-Domain Text-to-Speech Synthesis (2022, background): “Recently, non-autoregressive TTS models [23, 12, 18, 17] have been proposed and significantly speed up mel-spectrogram generation, and researchers have experimented on style modeling and transferring in parallel: Meta-StyleSpeech [31] generally adopts a speech encoding network for multi-speaker TTS ”