Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition
By Felix Wu, Kwangyoun Kim, Jing Pan, Kyu J. Han, Kilian Q. Weinberger, Yoav Artzi. 2021
137 citations recorded by the source corpus.
This paper is a study of performance-efficiency trade-offs in pre-trained models for automatic speech recognition (ASR). We focus on wav2vec 2.0, and formalize several architecture designs that influence both the model performance and its efficiency. Putting together all our observations, we introduce SEW-D (Squeezed and Efficient Wav2vec with Disentangled Attention), a pre-trained model architecture with significant improvements along both performance and efficiency dimensions across a variety of training setups. For example, under the 100h-960h semi-supervised setup on LibriSpeech, SEW-D achieves a 1.9x inference speedup compared to wav2vec 2.0, with a 13.5% relative reduction in word error rate. With a similar inference time, SEW reduces word error rate by 25–50% across different model sizes.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Minh-Thang Luong
- Quoc V. Le
- Mohit Bansal
- Yoshua Bengio
- Graham Neubig
- Kyunghyun Cho
- Hieu Pham
- Jianfeng Gao
Citations in the corpus
- On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “…unsupervised manner, has attracted attention from both the machine learning and the speech communities (Liu et al., 2019a; Feng et al., 2019; Nayak et al., 2019; Tjan-dra et al., 2019; Schneider et al., 2019; Baevski et al., 2020a; Chen and Hain, 2020; Morita and Koda, 2020; Tobing et al., 2020).”
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, neutral): “…processing (NLP; Peters et al. 2018; Radford et al. 2018; Devlin et al. 2019; Brown et al. 2020), speech processing (van den Oord et al., 2018; Schneider et al., 2019; Baevski et al., 2020b) as well as computer vision (Chen et al., 2020; 2021b; Caron et al., 2021; Bao et al., 2021; He et al.,…”
- Towards Domain-Agnostic Contrastive Learning (2020, background): “(Dai & Le, 2015; Howard & Ruder, 2018; Peters et al., 2018; Radford et al., 2019; Clark et al., 2020), and speech recognition (Schneider et al., 2019; Baevski et al., 2020).”
- Self-supervised Learning with Random-projection Quantizer for Speech Recognition (2022, unknown): “Self-supervised learning has shown impressive improvement for the quality of the speech recognition models in recent years (Schneider et al., 2019; Baevski et al., 2020a; 2019; 2020b; Hsu et al., 2021; Zhang et al., 2020; Chung et al., 2021; Zhang et al., 2021).”