ESPnet: End-to-End Speech Processing Toolkit
By Shinji Watanabe, Takaaki Hori, Shigeki Karita, Tomoki Hayashi, Jiro Nishitoba, Yuya Unno, Nelson Yalta, Jahn Heymann, Matthew Wiesner, Nanxin Chen and 2 more. 2018
1,730 citations recorded by the source corpus.
This paper introduces a new open source platform for end-to-end speech processing named ESPnet. ESPnet mainly focuses on end-to-end automatic speech recognition (ASR), and adopts widely-used dynamic neural network toolkits, Chainer and PyTorch, as a main deep learning engine. ESPnet also follows the Kaldi ASR toolkit style for data processing, feature extraction/format, and recipes to provide a complete setup for speech recognition and other speech processing experiments. This paper explains a major architecture of this software platform, several important functionalities, which differentiate ESPnet from other open source ASR toolkits, and experimental results with major ASR benchmarks.
Subfields: Artificial Intelligence, Speech & Audio, ML Systems & Hardware, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jinyu Li
- Shinji Watanabe
- Yoshua Bengio
- Jianfeng Gao
- Mohit Bansal
- Furu Wei
- Luke S. Zettlemoyer
- James R. Glass
In Speech & Audio
- Shinji Watanabe
- Jinyu Li
- Wei-Ning Hsu
- Alexei Baevski
- Gabriel Synnaeve
- Abdel-rahman Mohamed
- Yu Zhang
- Michael Auli
In ML Systems & Hardware
- Shinji Watanabe
- Nima Mesgarani
- Yoshua Bengio
- Michael Auli
- Kilian Q. Weinberger
- Adam Polyak
- Karen Livescu
- Marc Tommasi
In Signal Processing
Citations in the corpus
- GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10, 000 Hours of Transcribed Audio (2021, background): “This section describes the baseline systems and experimental results for four popular speech recognition toolkits, namely Athena, ESPnet [15], Kaldi [12] and Pika [16].”
- You Do Not Need More Data: Improving End-To-End Speech Recognition by Text-To-Speech Data Augmentation (2020, methodology): “As an ASR model, we chose the Transformer from the ESPnet(4) speech recognition toolkit [20], since it delivers close to state-of-the-art (SotA) results on LibriSpeech [21].”
- Transformers with convolutional context for ASR (2019, background): “Using convolutional layers as input processors before recurrent layers in acoustic encoders has been previously proposed for computational reasons with minimal impact on performance [17].”
- Wav2Letter++: A Fast Open-source Speech Recognition System (2018, methodology): “With the growing interest in automatic speech recognition (ASR), the open-source software ecosystem has seen a proliferation of ASR systems and toolkits, including Kaldi [1], ESPNet [2], OpenSeq2Seq [3] and Eesen[4].”
- Masked Language Model Scoring (2019, methodology): “, 2019) from ESPnet (Watanabe et al., 2018) on the 960hour LibriSpeech corpus (Panayotov et al.”
- Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion? (2019, positive): “Our baseline network follows the ESPnet architecture [25], with one encoder and two decoders: one based on connectionist temporal classification (CTC) and the other on an attention mechanism.”
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, methodology): “All baseline models, except for the publicly available VITS model on LibriTTS from ESPNet Toolkit [63], were official checkpoints released by the authors, including vocoders used in StyleTTS.”
- SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “We trim the waveform as ESPnet recipe (Watanabe et al., 2018).”
- WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “To simply implement such advanced models and obtain state-of-the-art reproducible results, researchers also release lots of open source toolkits, including Kaldi [12], Sphinx [13], Fariseq [14], ESPnet [15], and Wenet [16], etc.”
- Self-supervised Sequence-to-sequence ASR using Unpaired Speech and Text (2019, methodology): “ESPnet [28] is used to implement and execute all our experiments.”
- OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation (2021, methodology): “The compared methods include several API from Baidu and IBM companies and a state of the art model, Espnet [45], which is pretrained on the librispeech dataset, resulting 48.”
- A Vector Quantized Approach for Text to Speech Synthesis on Real-World Spontaneous Speech (2023, methodology): “We adopt ASR model 5 pretrained on GigaSpeech from ESPNet (Watanabe et al. 2018) for this purpose.”
- Speech Recognition via Large-Scale Weak Supervision (positive): “ESPnet (Watanabe et al. 2018, Hayashi et al. 2020).”
- Multilingual Sequence-to-Sequence Speech Recognition: Architecture, Transfer Learning, and Language Modeling (2018, methodology): “All experiments are implemented in ESPnet, end-to-end speech processing toolkit [29].”
- Improving Transformer-Based End-to-End Speech Recognition with Connectionist Temporal Classification and Language Model Integration (2019, methodology): “• We implement Transformer on the open ASR toolkit ESPnet [13] as a reproducible source code(1).”
- The AS-NU System for the M2VoC Challenge (2021, methodology): “The mel-spectrogram feature was extracted from an utterance using the ESPNet [16] toolkit.”