fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit

By Changhan Wang, Wei-Ning Hsu, Yossi Adi, Adam Polyak, Ann Lee, Peng-Jen Chen, Jiatao Gu, Juan Pino. 2021

35 citations recorded by the source corpus.

This paper presents fairseq Sˆ2, a fairseq extension for speech synthesis. We implement a number of autoregressive (AR) and non-AR text-to-speech models, and their multi-speaker variants. To enable training speech synthesis models with less curated data, a number of preprocessing tools are built and their importance is shown empirically. To facilitate faster iteration of development and analysis, a suite of automatic metrics is included. Apart from the features added specifically for this extension, fairseq Sˆ2 also benefits from the scalability offered by fairseq and can be easily integrated with other state-of-the-art systems provided in this framework. The code, documentation, and pre-trained models will be made available at https://github.com/pytorch/fairseq/tree/master/examples/speech_synthesis.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Wei-Ning Hsu
2. Yossi Adi
3. Gabriel Synnaeve
4. Juan Pino
5. Changhan Wang
6. Jiatao Gu
7. Alexis Conneau
8. Ann Lee

Citations in the corpus

• Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “Prior work uses labels to make the mapping between input (text and audio style) and output (speech) more deterministic to reduce underfitting [60, 46].”
• Textless Speech-to-Speech Translation on Real Data (2021, positive): “We use multi-speaker speech from the VoxPopuli ASR dataset (Wang et al., 2021c) text-to-unit (T2U) conversion is done with a Trans-former MT model (Vaswani et al., 2017) trained on single-speaker TTS data (described later) with characters as input and reduced orig-unit as target.”
• Textually Pretrained Speech Language Models (2023, methodology): “All SpeechLMs are optimized using a collection of publicly available academic speech datasets: LibriSpeech (LS) [Panayotov et al., 2015], LibriLight (LL) [Kahn et al., 2020], Spotify pod-casts [Clifton et al., 2020], People dataset [Galvez et al., 2021], and VoxPopuli [Wang et al., 2021a].”