Sequence-to-Sequence Speech Recognition with Time-Depth Separable Convolutions

By Awni Y. Hannun, Ann Lee, Qiantong Xu, Ronan Collobert. 2019

109 citations recorded by the source corpus.

We propose a fully convolutional sequence-to-sequence encoder architecture with a simple and efficient decoder. Our model improves WER on LibriSpeech while being an order of magnitude more efficient than a strong RNN baseline. Key to our approach is a time-depth separable convolution block which dramatically reduces the number of parameters in the model while keeping the receptive field large. We also give a stable and efficient beam search inference procedure which allows us to effectively integrate a language model. Coupled with a convolutional language model, our time-depth separable convolution architecture improves by more than 22% relative WER over the best previously reported sequence-to-sequence results on the noisy LibriSpeech test set.

Subfields: Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

Citations in the corpus

End-to-end ASR: from Supervised to Semi-Supervised Learning with Modern Architectures (2019, positive): “Time-Depth Separable (TDS) Convolution Acoustic Model We extend the TDS block (Hannun et al., 2019) (which is composed of one 2-D convolution layer and two fully-connected layers with ReLU, LayerNorm and residual connections in between), by increasing the number of channels in the feature maps spann”
VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretation (2021, methodology): “The ASR systems are TDS models (Hannun et al., 2019) trained with ASG criterion (Collobert et al.”
MLS: A Large-Scale Multilingual Dataset for Speech Research (2020, methodology): “For English, however, we use pre-trained model from [12] which uses TDS encoder and CTC loss on LibriSpeech [13] and pseudo labels from LibriVox .”
ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context (2020, methodology): “Depthwise separable convolutions [6] have been utilized to further increase the speed and accuracy of CNN models [7, 4].”
Libri-Light: A Benchmark for ASR with Limited or No Supervision (2019, positive): “(2) We use CTC to train small Melfilterbanks-based TDS systems[23], (7 TDS blocks, 20M parameters, total stride 2) on phonemes/graphemes respectively.”
Self-Training for End-to-End Speech Recognition (2019, methodology): “We follow the same training process as in [3] with soft-window pre-training and teacher-forcing with 20% dropout, 1% random sampling, 10% label smoothing and 1% word piece sampling for regularization.”
Semi-Supervised Speech Recognition via Local Prior Matching (2020, methodology): “Unless otherwise stated, we follow the recipe of Kahn et al. (2019a) which uses fewer TDS blocks in the encoder compared to Hannun et al. (2019) in order to generalize better when trained on the smaller LibriSpeech train-clean-100.”
Self-Training for End-to-End Speech Translation (2020, methodology): “Specifically, the model has 4 groups of TDS blocks with a 1-”
A Comparison of Transformer and LSTM Encoder Decoder Models for ASR (2019, methodology): “For faster training time, and inspired by [40], and similar to the Transformer, we also investigate a variant, where the decoder LSTM does only depend on the ground truth, instead of the attention context.”
Word-level Speech Recognition with a Dynamic Lexicon (2019, methodology): “Both our CTC and seq2seq architectures use a time-depth separable (TDS) convolution [16].”
Fast, Simpler and More Accurate Hybrid ASR Systems Using Wordpieces (2020, background): “These network architectures have evolved in recent years and can be broadly classiﬁed into two categories: 1) Those that support streaming during inference, such as time delay neural network (TDNN) [2], feed-forward sequential memory networks (FSMN) [3], long short-term memory (LSTM) [4], latency-co”
Quartznet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions (2019, methodology): “Hannun et al [12] applied a similar approach to ASR.”
End-to-End Speech Recognition: A Survey (2023, unknown): “Thisincludestheuseoffulllabelcontextdependencyboth forE2Esystems[51],[77],[174],[229],[262],[273],[274], [275],aswellasclassicalsystemsviafull-contextlanguage models[244],[245],[276],[277].”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “Specifically, we use the Wav2Letter++ [80] implementation of the seq2seq+TDS model [31], using the COCO caption text as the ground truth labels for ASR training.”
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning (2020, methodology): “This convolutional architecture is inspired by [22], but other acoustic models such as TimeDepth Separable Convolutions [24], VGG transformer [25], or ResDAVENet [12] could be used here.”