Very deep convolutional networks for end-to-end speech recognition

By Yu Zhang, William Chan, Navdeep Jaitly. 2016

420 citations recorded by the source corpus.

Sequence-to-sequence models have shown success in end-to-end speech recognition. However these models have only used shallow acoustic encoder networks. In our work, we successively train very deep convolutional networks to add more expressive power and better generalization for end-to-end ASR models. We apply network-in-network principles, batch normalization, residual connections and convolutional LSTMs to build very deep recurrent and convolutional structures. Our models exploit the spectral structure in the feature space and add computational depth without overfitting issues. We experiment with the WSJ ASR task and achieve 10.5% word error rate without any dictionary or language model using a 15 layer deep network.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Pretraining for Sequence to Sequence Learning (2016, background): “More recent acoustic models have found pretraining unnecessary (Xiong et al., 2016; Zhang et al., 2016; Chan et al., 2015), probably because the reconstruction objective of deep belief networks is too easy.”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “End-to-end speech recognition models have also been beneﬁted from the increases in model capacity [28, 29, 5].”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, background): “Convolution Neural Network (CNN) based models for end-toend (E2E) speech recognition is attracting an increasing amount of attention [1, 2, 3, 4].”
Towards Better Decoding and Language Model Integration in Sequence to Sequence Models (2016, background): “Label smoothing brings a large error rate reduction, nearly matching the performance achieved with very deep and sophisticated encoders [22].”
Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions (2019, background): “The encoder typically consists of several layers of large bidirectional LSTMs [9, 10].”
Residual LSTM: Design of a Deep Recurrent Architecture for Distant Speech Recognition (2017, methodology): “The initial design of residual LSTM was to simply add an input path to an LSTM output without scaling, which is similar to a ResLSTM block in [24].”
Syllable-Based Sequence-to-Sequence Speech Recognition with the Transformer in Mandarin Chinese (2018, neutral): “Sequence-to-sequence attention-based models have shown very encouraging results on English ASR tasks [1, 8, 10].”
Exploring Speech Enhancement with Generative Adversarial Networks for Robust Speech Recognition (2017, background): “The activations of the second layer are passed to a bidirectional, convolutional LSTM layer [29, 30], followed by three bidirectional LSTM layers.”
ESPnet: End-to-End Speech Processing Toolkit (2018, methodology): “, [33] does not use any language models, while [35] and [11] use a word-based language model through FST), we cannot directly compare them.”
Online and Linear-Time Attention by Enforcing Monotonic Alignments (2017, methodology): “…thank Jan Chorowski, Mark Daoust, Pietro Kreitlon Carolino, Dieterich Lawson, Navdeep Jaitly, George Tucker, Quoc V. Le, Kelvin Xu, Cinjon Resnick, Melody Guan, Matthew D. Hoffman, Jeffrey Dean, Kevin Swersky, Ashish Vaswani, and members of the Google Brain team for helpful discussions and insight.”
Jasper: An End-to-End Convolutional Neural Acoustic Model (2019, neutral): “seq2seq + deep conv [35] - 10.”
Optimal Completion Distillation for Sequence Learning (2018, methodology): “Our model is an attention-based seq2seq network with a deep convolutional frontend as used in Zhang et al. (2017).”
Transformer-Based Acoustic Modeling for Hybrid Speech Recognition (2019, background): “Convolutional embedding implicitly performs frame stacking as well as learns useful short-range spectral-temporal patterns [28].”
Mean Field Analysis of Neural Networks: A Law of Large Numbers (2018, neutral): “In speech recognition, neural networks are used to develop computer systems that automatically understand human speech (see [31], [3], [33], and [56]).”
Model Unit Exploration for Sequence-to-Sequence Speech Recognition (2019, methodology): “Thus, following [25], our encoder layers include two layers of 3×3 convolution with 32 channels with a stride of 2, which results in a total time reduction factor of 4.”
English Conversational Telephone Speech Recognition by Humans and Machines (2017, methodology): “Residual Networks were introduced for image recognition in [9] and used in speech recognition in [1, 10].”
Multifaceted protein–protein interaction prediction based on Siamese residual RCNN (2019, background): “It has been widely deployed in deep learning architectures for various tasks of image recognition (He et al., 2016), document classification (Conneau et al., 2017) and speech recognition (Zhang et al., 2017).”