Generative Pre-training for Speech with Flow Matching
By Alexander H. Liu, Matt Le, Apoorv Vyas, Bowen Shi, Andros Tjandra, Wei-Ning Hsu. 2023
72 citations recorded by the source corpus.
Generative models have gained more and more attention in recent years for their remarkable success in tasks that required estimating and sampling data distribution to generate high-fidelity synthetic data. In speech, text-to-speech synthesis and neural vocoder are good examples where generative models have shined. While generative models have been applied to different applications in speech, there exists no general-purpose generative model that models speech directly. In this work, we take a step toward this direction by showing a single pre-trained generative model can be adapted to different downstream tasks with strong performance. Specifically, we pre-trained a generative model, named SpeechFlow, on 60k hours of untranscribed speech with Flow Matching and masked conditions. Experiment results show the pre-trained generative model can be fine-tuned with task-specific data to match or surpass existing expert models on speech enhancement, separation, and synthesis. Our work suggested a foundational model for generation tasks in speech can be built with generative pre-training.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Apoorv Vyas
- Bowen Shi
- Matt Le
- Andros Tjandra
- Yi-Chiao Wu
- Baishan Guo
- Jiemin Zhang
- Xinyue Zhang
Citations in the corpus
- Audiobox: Unified Audio Generation with Natural Language Prompts (2023, methodology): “This work is heavily built upon the training objective and model architecture of Voicebox (Le et al., 2023), and the self-supervised objective of SpeechFlow (Liu et al., 2023a).”