Conformer: Convolution-augmented Transformer for Speech Recognition
By Anmol Gulati, James Qin, Chung-Cheng Chiu, Niki Parmar, Yu Zhang, Jiahui Yu, Wei Han, Shibo Wang, Zhengdong Zhang, Yonghui Wu and 1 more. 2020
4,195 citations recorded by the source corpus.
Recently Transformer and Convolution neural network (CNN) based models have shown promising results in Automatic Speech Recognition (ASR), outperforming Recurrent neural networks (RNNs). Transformer models are good at capturing content-based global interactions, while CNNs exploit local features effectively. In this work, we achieve the best of both worlds by studying how to combine convolution neural networks and transformers to model both local and global dependencies of an audio sequence in a parameter-efficient way. To this regard, we propose the convolution-augmented transformer for speech recognition, named Conformer. Conformer significantly outperforms the previous Transformer and CNN based models achieving state-of-the-art accuracies. On the widely used LibriSpeech benchmark, our model achieves WER of 2.1%/4.3% without using a language model and 1.9%/3.9% with an external language model on test/testother. We also observe competitive performance of 2.7%/6.3% with a small model of only 10M parameters.
Subfields: Artificial Intelligence, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yoshua Bengio
- Quoc V. Le
- Mohit Bansal
- Jianfeng Gao
- Percy Liang
- Luke S. Zettlemoyer
- Jinyu Li
- Graham Neubig
In Speech & Audio
- Jinyu Li
- Wei-Ning Hsu
- Yu Zhang
- Shinji Watanabe
- Tara N. Sainath
- James R. Glass
- Gabriel Synnaeve
- Hung-yi Lee
In Signal Processing
Citations in the corpus
- Rethinking Attention with Performers (2020, background): “Several solutions have been proposed to address this issue (Beltagy et al., 2020; Gulati et al., 2020; Chan et al., 2020; Child et al., 2019; Bello et al., 2019).”
- wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, neutral): “1 Conformer [15] LSTM 2.”
- FLEURS: FEW-Shot Learning Evaluation of Universal Representations of Speech (2022, background): “Speech technology has been rapidly evolved in the past few years, with striking achievements nourished from selfattention models [3, 4], pre-training approaches [5, 6, 7, 2], and massively multilingual speech models [8, 9].”
- mSLAM: Massively multilingual joint pre-training for speech and text (2022, methodology): “Multimodal pre-training: SLAM (Bapna et al., 2021) is a multimodal speech and text pretraining method, which trains a single Conformer (Gulati et al., 2020) with SpanBERT (Joshi et al., 2020) and w2v-BERT (Chung et al., 2021) self-supervised losses that leverage unlabeled monomodal data, as well as ”
- BigSSL: Exploring the Frontier of Large-Scale Semi-Supervised Learning for Automatic Speech Recognition (2021, methodology): “We have used Conformers [32] as the encoder architecture in”
- GigaSpeech: An Evolving, Multi-domain ASR Corpus with 10, 000 Hours of Transcribed Audio (2021, methodology): “The ESPnet baseline uses conformer (Convolution-Augmented Transformer) [7], which is a recently proposed architecture combining the local sensitivies of convolutional neural networks with the long-range interactions of transformers [20].”
- Unsupervised Speech Recognition (2021, background): “, 2015) has seen rapid improvement over the last few years due to advances in model architectures (Dong et al., 2018; Synnaeve et al., 2020; Gulati et al., 2020), semisupervised learning (Xu et al.”
- SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, positive): “In our implementation, SpeechStew uses the Conformer [29] RNN-T [30] architecture.”
- Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, neutral): “SOTA: LS [44] / SB [45] / CV [8] / TD [46] - - 1.”
- Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Various model architectures [6, 7, 8, 9, 10] and training schemes [11, 12, 13] were proposed.”
- Self-Training and Pre-Training are Complementary for Speech Recognition (2020, background): “Speech recognition models trained on labeled speech data has progressed substantially in the recent past [1, 2, 3, 4].”
- Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “Method No LM With LM dev dev-other test test-other dev dev-other test test-other Baseline Conformer L [5] 1.”
- Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “S2T extension FAIRSEQ S2T adds attention-based RNN models (Chan et al., 2016; B ´ erard et al., 2018), Transformer models (Vaswani et al., 2017; Mohamed et al., 2019) as well as the latest Con-former models (Gulati et al., 2020) for ASR and ST.”
- Primer: Searching for Efficient Transformers for Language Modeling (2021, methodology): “Although depthwise convolutions have been used for Transformers before [41, 42], using them after each dense head projection has not been done to the best of our knowledge.”
- Scaling Autoregressive Models for Content-Rich Text-to-Image Generation (2022, background): “Parti is a sequence-to-sequence model based on the Transformer [4], an architecture critical to performance on many tasks, including machine translation [4], speech recognition [22, 23], conversational modeling [24], image captioning [25], and many others.”