Improvements to Deep Convolutional Neural Networks for LVCSR

By Tara N. Sainath, Brian Kingsbury, Abdel-rahman Mohamed, George E. Dahl, George Saon, Hagen Soltau, Tomás Beran, Aleksandr Y. Aravkin, Bhuvana Ramabhadran. 2013

239 citations recorded by the source corpus.

Deep Convolutional Neural Networks (CNNs) are more powerful than Deep Neural Networks (DNN), as they are able to better reduce spectral variation in the input signal. This has also been confirmed experimentally, with CNNs showing improvements in word error rate (WER) between 4-12% relative compared to DNNs across a variety of LVCSR tasks. In this paper, we describe different methods to further improve CNN performance. First, we conduct a deep analysis comparing limited weight sharing and full weight sharing with state-of-the-art features. Second, we apply various pooling strategies that have shown improvements in computer vision to an LVCSR speech task. Third, we introduce a method to effectively incorporate speaker adaptation, namely fMLLR, into log-mel features. Fourth, we introduce an effective strategy to use dropout during Hessian-free sequence training. We find that with these improvements, particularly with fMLLR and dropout, we are able to achieve an additional 2-3% relative improvement in WER on a 50-hour Broadcast News task over our previous best CNN baseline. On a larger 400-hour BN task, we find an additional 4-5% relative improvement over our previous best CNN baseline.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dong Yu
2. Jinyu Li
3. Yoshua Bengio
4. Li Deng
5. Tara N. Sainath
6. Yu Zhang
7. Quoc V. Le
8. Shinji Watanabe

In Signal Processing

1. Dong Yu
2. Jinyu Li
3. Li Deng
4. Tara N. Sainath
5. Yoshua Bengio
6. Shinji Watanabe
7. Yu Zhang
8. Florian Metze

Citations in the corpus

• Recent advances in convolutional neural networks (2015, methodology): “Besides, small filters have successfully applied in acoustic modeling in hybrid NN-HMM ASR system, and pooling operations are replaced by densely connected layers for ASR tasks [320].”
• Deep Learning: Methods and Applications (2014, methodology): “[306] combined dropout with a number of novel techniques described in this section (including the use of deep CNNs, Hessian-free sequence learning, the use of ReLU units, and the use of joint fMLLR and filterbank features, etc.”
• Very deep convolutional networks for end-to-end speech recognition (2016, methodology): “CNNs use shared weight filters and pooling to give the model better spectral and temporal invariance properties, thus typically yield better generalized and more robust models compared to DNNs [13].”
• Joint training of convolutional and non-convolutional neural networks (2014, methodology): “Closest to our work is the work described in [7], which mentioned a combination of MLPs and CNNs.”
• Data Augmentation for Deep Neural Network Acoustic Modeling (2015, methodology): “2 shows a mean and speaker normalized LogMel feature extraction pipeline used in IBM CNN systems [31][32].”
• Towards End-to-End Speech Recognition with Deep Convolutional Neural Networks (2016, methodology): “Following the suggestion from [4], we only perform pooling along the frequency band on the first convolutional layer.”
• A review of convolutional neural networks in computer vision (2024, unknown): “In addition, the fully connected layer can integrate local information that is class-dis-tinctive in the convolution or pooling layers (Sainath et al. 2013).”
• Deep Learning for Audio Signal Processing (2019, background): “In addition to the great success of deep feedforward and convolutional networks [91], LSTMs and GRUs have been shown to outperform feedforward DNNs [92].”
• Recommendation system based on deep learning methods: a systematic review and new directions (2019, unknown): “With the emergence of the deep neural network, there has been a remarkable achievement in many application fields such as computer visions (Sainath et al. 2013), NLP (Poria et al.”
• Adan: Adaptive Nesterov Momentum Algorithm for Faster Optimizing Deep Models (2022, background): “computer vision [7, 22 8, 14–16] and natural language processing [17, 18].”
• On speaker adaptation of long short-term memory recurrent neural networks (2015, methodology): “Third, the acoustic model is trained (and decoded) using speaker-adaptive (SA) features [3, 21, 22] or features enriched with SD information [23, 24, 25, 26].”
• Multi-GPU Training of ConvNets (2013, background): “Convolutional neural networks [3] have proven useful in many domains, including computer vision [1, 4, 5], audio processing [6, 7] and natural language processing [8].”
• Building DNN acoustic models for large vocabulary speech recognition (2014, methodology): “parameters on both 50 and 400 hour broadcast news LVCSR tasks [28].”
• Deep Speech: Scaling up end-to-end speech recognition (2014, background): “Researchers have also begun choosing designs that map well to GPU hardware to gain even more efficiency, including convolutional [23, 4, 35] and locally connected [7, 5] networks, especially when optimized libraries like cuDNN [2] and BLAS are available.”
• Convolutional Neural Networks for Distant Speech Recognition (2014, background): “Channelwise convolution may also be viewed as a special case of 2- dimensional convolution, where the effective pooling region is determined in frequency but varies in time depending on the actual time delays between the microphones.”
• MICN: Multi-scale Local and Global Context Modeling for Long-term Series Forecasting (2023, background): “1 CNNS AND TRANSFORMERS Convolutional neural networks (CNN) are widely used in computer vision, natural language processing and speech recognition (Sainath et al., 2013) (Li et al.”