Recent Developments on Espnet Toolkit Boosted By Conformer

By Pengcheng Guo, Florian Boyer, Xuankai Chang, Tomoki Hayashi, Yosuke Higuchi, Hirofumi Inaguma, Naoyuki Kamo, Chenda Li, Daniel Garcia-Romero, Jiatong Shi and 5 more. 2020

292 citations recorded by the source corpus.

In this study, we present recent developments on ESPnet: End-to- End Speech Processing toolkit, which mainly involves a recently proposed architecture called Conformer, Convolution-augmented Transformer. This paper shows the results for a wide range of end- to-end speech processing applications, such as automatic speech recognition (ASR), speech translations (ST), speech separation (SS) and text-to-speech (TTS). Our experiments reveal various training tips and significant performance benefits obtained with the Conformer on different tasks. These results are competitive or even outperform the current state-of-art Transformer models. We are preparing to release all-in-one recipes using open source and publicly available corpora for all the above tasks with pre-trained models. Our aim for this work is to contribute to our research community by reducing the burden of preparing state-of-the-art research environments usually requiring high resources.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10, 000 Hours of Transcribed Audio (2021, methodology): “We use the implementation provided in the ESPnet toolkit [21].”
A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, positive): “Conformer Given the recent success of Convolution-augmented Transformer (Conformer) on various speech tasks (Gulati et al., 2020; Guo et al., 2021), we adopt Conformer as the backbone of our encoder and decoder.”
WENETSPEECH: A 10000+ Hours Multi-Domain Mandarin Corpus for Speech Recognition (2021, methodology): “ESPnet Benchmark(6) The ESPnet baseline employs a Conformer model [11, 33] which is designed to capture the global context with the multiheaded selfattention module and learn the local correlations synchronously with the convolution module.”
Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “It is tempting to use a non-autoregressive architecture for the synthesizer of Translatotron 2, which may signiﬁcantly reduce its inference latency, similar to recent works on non-autoregressive TTS (Ren et al., 2019; 2021; Guo et al., 2021; Lee et al., 2021b; Elias et al., 2021b;a).”
ISTFTNET: Fast and Lightweight Mel-Spectrogram Vocoder Incorporating Inverse Short-Time Fourier Transform (2022, methodology): “Next, we examine the effectiveness of our approach when applied to TTS synthesis, focusing on the difference in performance between V1 and V1-C8C8I when combined with Conformer-FS2 [38] (a combination of Conformer [39] and FastSpeech 2 (FS2) [5]).”
End-to-End Speech Recognition: A Survey (2023, background): “Both Trans-former and Conformer have shown competitive performance to LSTMs on many tasks [102], [103].”
Translatotron 2: Robust direct speech-to-speech translation (2021, methodology): “C N ON - AUTOREGRESSIVE SYNTHESIZER Following recent non-autoregressive TTS works (Ren et al., 2021; Guo et al., 2021; Lee et al., 2021; Elias et al., 2021a), we explored using non-autoregressive synthesizer in Translatotron 2, which may have signiﬁcantly lower latency at inference time.”
Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “Although having higher model complexity than Transformer, Conformer has been reported to outperform Transformer in various tasks [97, 98].”
TranSpeech: Speech-to-Speech Translation With Bilateral Perturbation (2022, background): “The conformer model (Guo et al., 2021; Chen et al., 2021) has demonstrated its efﬁciency in combining convolution neural networks and transformers to model both local and global dependencies of audio in a parameter-efﬁcient way, achieving state-of-the-art results on various downstream tasks.”
Nonparallel Expressive TTS for Unseen Target Speaker using Style-Controlled Adaptive Layer and Optimized Pitch Embedding (2023, methodology): “ CFS2-PWGAN: We trained a Conformer (Convolution-augmented Transformer) based on FastSpeech2 [31] with a Parallel WaveGAN model [32].”
Opencpop: A High-Quality Open Source Chinese Popular Song Corpus for Singing Voice Synthesis (2022, methodology): “To face this limitation, Conformer thus came into being and has shown promising performance in the TTS task [36].”