ESPnet-ST: All-in-One Speech Translation Toolkit
By Hirofumi Inaguma, Shun Kiyono, Kevin Duh, Shigeki Karita, Nelson Yalta, Tomoki Hayashi, Shinji Watanabe. 2020
174 citations recorded by the source corpus.
We present ESPnet-ST, which is designed for the quick development of speech-to-speech translation systems in a single framework. ESPnet-ST is a new project inside end-to-end speech processing toolkit, ESPnet, which integrates or newly implements automatic speech recognition, machine translation, and text-to-speech functions for speech translation. We provide all-in-one recipes including data pre-processing, feature extraction, training, and decoding pipelines for a wide range of benchmark datasets. Our reproducible results can match or even outperform the current state-of-the-art performances; these pre-trained models are downloadable. The toolkit is publicly available at https://github.com/espnet/espnet.
Subfields: Artificial Intelligence, Machine Translation, Speech & Audio
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Machine Translation
In Speech & Audio
- Juan Pino
- Changhan Wang
- Jinyu Li
- Junichi Yamagishi
- Mingxuan Wang
- Rong Ye
- Anne Wu
- Sakriani Watiasri Sakti
Citations in the corpus
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “Compared with previous Transformer-based approaches (Di Gangi et al., 2019b; Inaguma et al., 2020), our bilingual models achieve comparative results to the state of the art without applying additional techniques such as speed perturbation and pre-trained decoder from MT.”
- A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, positive): “For speech-editing systems and prompt-based TTS, we use the publicly available duration predictor from FastSpeech 2 implemented in ESPnet (Inaguma et al., 2020).”
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “ESPnet ST (Inaguma et al., 2020), and Adapter Tuning (Le et al.”
- Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation (2021, positive): “Our basic Transformer-based E2E-ST framework has similar settings with ESPnet-ST(Inaguma et al., 2020). the speech input is first down-sampled the speech input with 2 layers of 2D convolution of size 3 with stride size of 2.”
- GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, methodology): “With the success of attention-based models for speech and text-related tasks, a typical and effective baseline model for ST is speech-transformer [2, 3, 4, 5], which has much of the same model structure as the commonly used MT model Transformer [6], except for the pre-processing down-sampling module”
- Transformer-Based Direct Speech-To-Speech Translation with Transcoder (2021, methodology): “reported that a Transformer-based model outperformed an RNN-based model on end-to-end ASR, MT, and TTS tasks [8].”
- A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks (2020, background): “Attention-based encoder-decoder modeling is a natural and powerful paradigm for speech to text tasks, such as automatic speech recognition (ASR) and speech translation (ST), and that has led to significant progress [1, 2, 3, 4, 5, 6].”
- Lightweight Adapter Tuning for Multilingual Speech Translation (2021, methodology): “For speech applications (Inaguma et al., 2020; Wang et al., 2020), the embedding layer of the encoder is often a small convolutional neural network (Fukushima and Miyake, 1982; LeCun et al.”
- Learning Shared Semantic Space for Speech-to-Text Translation (2021, unknown): “, 2016), solving Speech Translation in an end-to-end manner has attracted extensive attention (Vila et al., 2018; Salesky et al., 2018, 2019; Di Gangi et al., 2019b; Bahar et al., 2019a; Di Gangi et al., 2019c; Inaguma et al., 2020).”
- Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigm (2020, methodology): “We used the ESPNet[18] Kaldi recipe for CMU AN4 ASR [19] and a single-layer LSTM model with 64 units, trained to 100 epochs, selected the best model based on CTC loss without attention (mtlalpha = 1.0), and decoded with beam size 20.”
- Investigating Self-Supervised Pre-Training for End-to-End Speech Translation (2020, methodology): “Finally, in order to compare ourselves with the stateof-the-art [28], we decode How2 dev5 (a.”
- Leveraging Pseudo-labeled Data to Improve Direct Speech-to-Speech Translation (2022, methodology): “We train a Transformer-based E2E ST following the model setting in [27] with ASR pre-training.”
- DUB: Discrete Unit Back-translation for Speech Translation (2023, unknown): “…32.8 27.5 27.8 Speechformer (Papi et al., 2021) 23.6 - 28.5 - Cascaded (Inaguma et al., 2020) 23.6 33.8 28.7 28.7 MTL (Tang et al., 2021) 23.9 33.1 28.6 28.5 Self-training (Pino et al., 2020) 25.2 34.5 - - SpeechT5 (Ao et al., 2022) 25 (Wang et al., 2020; Inaguma et al., 2020; Zhao et al., 2021).”