ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context

By Wei Han, Zhengdong Zhang, Yu Zhang, Jiahui Yu, Chung-Cheng Chiu, James Qin, Anmol Gulati, Ruoming Pang, Yonghui Wu. 2020

302 citations recorded by the source corpus.

Convolutional neural networks (CNN) have shown promising results for end-to-end speech recognition, albeit still behind other state-of-the-art methods in performance. In this paper, we study how to bridge this gap and go beyond with a novel CNN-RNN-transducer architecture, which we call ContextNet. ContextNet features a fully convolutional encoder that incorporates global context information into convolution layers by adding squeeze-and-excitation modules. In addition, we propose a simple scaling method that scales the widths of ContextNet that achieves good trade-off between computation and accuracy. We demonstrate that on the widely used LibriSpeech benchmark, ContextNet achieves a word error rate (WER) of 2.1%/4.6% without external language model (LM), 1.9%/4.1% with LM and 2.9%/7.0% with only 10M parameters on the clean/noisy LibriSpeech test sets. This compares to the previous best published system of 2.0%/4.6% with LM and 3.9%/11.3% with 20M parameters. The superiority of the proposed ContextNet model is also verified on a much larger internal dataset.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “1 for ContextNet [17], the baseline architecture of the state of the art [42].”
Multitask Training with Text Data for End-to-End Speech Recognition (2020, background): “Various model architectures [6, 7, 8, 9, 10] and training schemes [11, 12, 13] were proposed.”
Self-Training and Pre-Training are Complementary for Speech Recognition (2020, neutral): “Speech recognition models trained on labeled speech data has progressed substantially in the recent past [1, 2, 3, 4].”
Pushing the Limits of Semi-Supervised Learning for Automatic Speech Recognition (2020, methodology): “As one of the baselines, we train a Conformer L model with NST without pre-training, using ContextNets [20] as intermediate models.”
Conformer: Convolution-augmented Transformer for Speech Recognition (2020, methodology): “Our 10M model shows an improvement when compared to similar sized contemporary work [10] with 2.”
Scaling End-to-End Models for Large-Scale Multilingual ASR (2021, methodology): “To justify the effectiveness of Conformer encoders for multilingual modeling, we compare three encoders with the same LSTM-based decoder: (1) LSTM encoder with 8 layers, 2048D hidden units and 640D output units; (2) ContextNet encoder with 24 layers, 640D hidden units and channel scale of 2 [40]; an”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, methodology): “, 2013), RNN transducers (RNN-T; Graves, 2012; Han et al., 2020; Gulati et al., 2020), and sequence-to-sequence (a.”
SpeechBrain: A General-Purpose Speech Toolkit (2021, methodology): “Our best system is a transformer [61] combined with a convolutional front-end based on ContextNet [83].”
End-to-End Speech Recognition: A Survey (2023, background): “Such architectures include convolution-based architectures, such as ContextNet [97].”
Universal ASR: Unify and Improve Streaming ASR with Full-context Modeling (2020, methodology): “Convolution layers, however, convolve feature across its neighbor timesteps within a ﬁxed window ( e.g ., kernel size is 3, 5, or larger), and has been widely used in sequence modeling (Gehring et al., 2017; Han et al., 2020; Gulati et al., 2020).”
Large-Scale End-to-End Multilingual Speech Recognition and Language Identification with Multi-Task Learning (2020, methodology): “End-to-end Automatic Speech Recognition (ASR) methods have demonstrated favorable results compared to conventional Hidden Markov Model (HMM) methods [1, 2].”
slimIPL: Language-Model-Free Iterative Pseudo-Labeling (2020, unknown): “ContexNet [79] 80ms 1k wp CNN-RNN-T - - 2.”
MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (2023, background): “Convolutional neural networks (CNN) are widely used in computer vision, natural language processing and speech recognition (Sainath et al., 2013) (Li et al., 2019a) (Han et al., 2020).”
Bridging the gap between streaming and non-streaming ASR systems bydistilling ensembles of CTC and RNN-T models (2021, background): “Such behavior has already been observed in the experiments of [26], where the authors compare two different student models for the same ContextNet teacher [27] in a noisy student setting [28].”