Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks

By Y. Zhang, Mohammad Pezeshki, Philemon Brakel, Saizheng Zhang, César Laurent, Yoshua Bengio, Aaron C. Courville. 2016

387 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) are effective models for reducing spectral variations and modeling spectral correlations in acoustic features for automatic speech recognition (ASR). Hybrid speech recognition systems incorporating CNNs with Hidden Markov Models/Gaussian Mixture Models (HMMs/GMMs) have achieved the state-of-the-art in various benchmarks. Meanwhile, Connectionist Temporal Classification (CTC) with Recurrent Neural Networks (RNNs), which is proposed for labeling unsegmented sequences, makes it feasible to train an end-to-end speech recognition system instead of hybrid settings. However, RNNs are computationally expensive and sometimes difficult to train. In this paper, inspired by the advantages of both CNNs and the CTC approach, we propose an end-to-end speech framework for sequence labeling, by combining hierarchical CNNs with CTC directly without recurrent connections. By evaluating the approach on the TIMIT phoneme recognition task, we show that the proposed model is not only computationally efficient, but also competitive with the existing baseline systems. Moreover, we argue that CNNs have the capability to model temporal correlations with appropriate context information.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

In Signal Processing

Citations in the corpus

Fully Convolutional Speech Recognition (2018, methodology): “They have more recently been used in end-to-end frameworks, but only in combination with recurrent layers [7], or n-gram language models [12], or for phone recognition [13, 14].”
Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, methodology): “Our research builds on prior work that has explored using time-delay neural networks (TDNN), other forms of convolutional neural networks, and Connectionist Temporal Classification (CTC) loss [5, 6, 7].”
Puzzle Mix: Exploiting Saliency and Local Statistics for Optimal Mixup (2020, background): “Second, in order to respect the underlying local statistics of the data (Huang & Mumford, 1999; Zhang et al., 2017; Smith, 2003), we consider the following criteria.”
Neural Speech Recognizer: Acoustic-to-Word LSTM Model for Large Vocabulary Speech Recognition (2016, neutral): “In the past, the best speech recognition systems have used many complex modeling techniques for improving accuracy: for example the use of hand-crafted feature representations, speaker or environment adaptation with feature or afﬁne transformations, and context-dependent (CD) phonetic models with…”
Self-attention Networks for Connectionist Temporal Classification in Speech Recognition (2019, methodology): “As this hinders parallelization, later works proposed partially- or purely-convolutional CTC models [8–11] and convolution-heavy encoder-decoder models [16] for ASR.”
On Using Backpropagation for Speech Texture Generation and Voice Conversion (2017, methodology): “To apply the texture generation and stylization techniques to speech we train a fully convolutional speech recognition network following [19] on the Wall Street Journal dataset.”
Stochastic Precision Ensemble: Self-Knowledge Distillation for Quantized Deep Neural Networks (2020, positive): “Deep neural networks (DNNs) have achieved remarkable accuracy for tasks in a wide range of applications, including image processing (He et al. 2016a), machine translation (Gehring et al. 2017), and speech recognition (Zhang et al. 2017).”
Accurate and Efficient 2-bit Quantized Neural Networks (2019, background): “Deep Convolutional Neural Networks (CNNs) have achieved remarkable accuracy for tasks in a wide range of application domains including image processing (He et al., 2016b), machine translation (Gehring et al., 2017), and speech recognition (Zhang et al., 2017).”
Who Needs Words? Lexicon-Free Speech Recognition (2019, background): “However, most competitive ASR systems, character based or not, use a beam search decoder constrained on a word-level language model and lexicon [14, 15, 16, 17].”
Advances in Joint CTC-Attention Based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM (2017, methodology): “Our encoder network is boosted by using deep CNN, which is motivated by the prior studies [16, 15].”
Robust Laughter Detection in Noisy Environments (2021, background): “Though work over the last few years on many audio detection problems has demonstrated that features learned from spectrograms using deep convolutional networks, paired with data augmentation, significantly outperform previous approaches based on MFCC’s [16, 17, 18], the audio classification methods ”
Deep Audio-Visual Speech Recognition (2018, methodology): “For example, fully convolutional networks have been used for ASR with CTC [51], [55] or a simplified vari-”
Quaternion Convolutional Neural Networks for End-to-End Automatic Speech Recognition (2018, methodology): “Furthermore, QCNNs always performed better than CNNs independently of the model topologies.”
PACT: Parameterized Clipping Activation for Quantized Neural Networks (2018, background): “Deep Convolutional Neural Networks (CNNs) have achieved remarkable accuracy for tasks in a wide range of application domains including image processing (He et al. (2016b)), machine translation (Gehring et al. (2017)), and speech recognition (Zhang et al. (2017)).”
Deep Learning for Magnetic Resonance Fingerprinting: A New Approach for Predicting Quantitative Parameter Values from Time Series (2017, methodology): “The architecture of the network (Figure 1) was inspired by neural networks used in the domain of speech processing [12], as these problems are”