Advances in Very Deep Convolutional Neural Networks for LVCSR

By Tom Sercu, Vaibhava Goel. 2016

44 citations recorded by the source corpus.

Very deep CNNs with small 3x3 kernels have recently been shown to achieve very strong performance as acoustic models in hybrid NN-HMM speech recognition systems. In this paper we investigate how to efficiently scale these models to larger datasets. Specifically, we address the design choice of pooling and padding along the time dimension which renders convolutional evaluation of sequences highly inefficient. We propose a new CNN design without timepadding and without timepooling, which is slightly suboptimal for accuracy, but has two significant advantages: it enables sequence training and deployment by allowing efficient convolutional evaluation of full utterances, and, it allows for batch normalization to be straightforwardly adopted to CNNs on sequence data. Through batch normalization, we recover the lost peformance from removing the time-pooling, while keeping the benefit of efficient convolutional evaluation. We demonstrate the performance of our models both on larger scale data than before, and after sequence training. Our very deep CNN model sequence trained on the 2000h switchboard dataset obtains 9.4 word error rate on the Hub5 test-set, matching with a single model the performance of the 2015 IBM system combination, which was the previous best published result.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Jinyu Li
2. Dong Yu
3. Li Deng
4. Yu Zhang
5. Tara N. Sainath
6. Yoshua Bengio
7. William Chan
8. Shinji Watanabe

Citations in the corpus

• Very deep convolutional networks for end-to-end speech recognition (2016, background): “Recently, very deep CNNs architectures [15] have also been shown to be successful in ASR [16, 17], using more non-linearities, but fewer parameters.”
• English Conversational Telephone Speech Recognition by Humans and Machines (2017, background): “Padding along the time direction would modify the values on the edges based on the input sliding window location, thus making efficient convolution over a full utterance impossible (see [23]).”
• Knowledge distillation across ensembles of multilingual models for low-resource languages (2017, methodology): “The VGG model [26, 34, 35, 36] comprises 12 convolutional layers, with a max-pooling layer inserted after every 3 convolutional layers, followed by 5 fully connected layers.”
• End-to-End attention based text-dependent speaker verification (2016, methodology): “The CNN model in the end-to-end system uses zero padding and 1-step striding during the convolution operation [18].”
• Dense Prediction on Sequences with Time-Dilated Convolutions for Speech Recognition (2016, methodology): “This allows to combine the performance gains of pooling [7], while maintaining the computational efficiency and ability to apply batch normalization [8].”
• Residual Convolutional CTC Networks for Automatic Speech Recognition (2017, background): “To the end, deep CNN (about 10 convolutional layers) showed great performance in noisy speech recognition (Qian & Wood-land, 2016; Sercu & Goel, 2016).”