Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq

By Changhan Wang, Yun Tang, Xutai Ma, Anne Wu, Dmytro Okhonko, Juan Pino. 2020

334 citations recorded by the source corpus.

We introduce fairseq S2T, a fairseq extension for speech-to-text (S2T) modeling tasks such as end-to-end speech recognition and speech-to-text translation. It follows fairseq’s careful design for scalability and extensibility. We provide end-to-end workflows from data pre-processing, model training to offline (online) inference. We implement state-of-the-art RNN-based as well as Transformer-based models and open-source detailed training recipes. Fairseq’s machine translation models and language models can be seamlessly integrated into S2T workflows for multi-task learning or transfer learning. Fairseq S2T is available at https://github.com/pytorch/fairseq/tree/master/examples/speech_to_text.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

Citations in the corpus

ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesis (2021, background): “The success of this approach is evident in domains as diverse as language modeling [5], music generation [14], neural machine translation [43, 73], and (conditional) image synthesis [51, 6].”
Robust Speech Recognition via Large-Scale Weak Supervision (2022, methodology): “We attribute this to the 68,000 hours of X → en translation data for these languages in our pre-training dataset which, although noisy, is vastly larger than the 861 hours of training data for X → en translation in CoVoST2.”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “For Section 4.3.2, we train Transformer models with cross-entropy criterion using fairseq S2T (Ott et al., 2019; Wang et al., 2020a).”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “We use the default training setting of speech translation in Fairseq ST (Wang et al., 2020), and we also average the last 10 checkpoints and use a beam size of 5 for decoding.”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “We apply the same training for Es but find that a supervised ASR model with the s2t transformer l architecture in FAIRSEQ is better.”
Textless Speech-to-Speech Translation on Real Data (2021, positive): “All experiments are conducted using fairseq (Ott et al., 2019; Wang et al., 2020a, 2021b) 4 .”
Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “The majority of the speech translation research has focused on the S2T setup.”
Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “Previous SOTAs: We compare to the best end-to-end (E2E) model from previous literature (Wang et al., 2020b; Iranzo-S´anchez et al., 2020) on each translation direction, which is usually the best-performing multilingual model trained with parallel data from all directions (both X-En and En-X) and…”
GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, methodology): “With the success of attention-based models for speech and text-related tasks, a typical and effective baseline model for ST is speech-transformer [2, 3, 4, 5], which has much of the same model structure as the commonly used MT model Transformer [6], except for the pre-processing down-sampling module”
Lightweight Adapter Tuning for Multilingual Speech Translation (2021, methodology): “Our implementation is based on the FAIRSEQ S2T toolkit (Wang et al., 2020).”
Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, methodology): “We use the “T-Md” conﬁguration from (Wang et al., 2020a) in all experiments.”
DUB: Discrete Unit Back-translation for Speech Translation (2023, methodology): “Fairseq ST (Wang et al., 2020) 22.7 32.9 27.2 27.6 NeurST (Zhao et al., 2021) 22.8 33.3 27.4 27.8 Espnet ST (Inaguma et al., 2020) 22.9 32.8 28.0 27.9 E2E-ST-JT (Du et al., 2022) 23.1 32.8 27.5 27.8 Speechformer (Papi et al., 2021) 23.6 - 28.5 - Cascaded (Inaguma et al., 2020) 23.6 33.8 28.7 28.7…”
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, methodology): “…target units, and then transcribe the speech to text and compute BELU; 3) S2T+TTS cascaded system, where we train the S2T basic transformer model (Wang et al., 2020a) and then apply TTS model (Ren et al., 2020; Kong et al., 2020) for speech generation; 4) basic transformer (Lee et al., 2021a)…”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, methodology): “Another strong multilingual s2t system, E2E S2T, was proposed in Wang et al. (2020a) for evaluation on the CoVoST2 dataset.”
The Multilingual TEDx Corpus for Speech Recognition and Translation (2021, methodology): “We used FAIRSEQ [32] to train end-to-end Transformer models for ST, using 80-dimensional log mel filterbank features with cmvn and SpecAugment [50], and 1-d convolutions for downsampling of speech in time.”
Direct simultaneous speech to speech translation (2021, methodology): “We propose variational monotonic multihead attention (V-MMA), which models the alignment with the latent variable α instead of recurrent estimation in sequence-to-sequence modeling.”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “Fairseq Models FAIRSEQ provides a collection of MT (Ng et al., 2019), ST (Wang et al., 2020), unsupervised speech pre-training and ASR (Baevski et al., 2020b; Hsu et al., 2021) models that demonstrate state-of-the-art performance on standard benchmarks.”
A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks (2020, background): “The models are trained with Fairseq [6].”