Deep Convolutional Neural Networks for Large-scale Speech Tasks

By Tara N. Sainath, Brian Kingsbury, George Saon, Hagen Soltau, Abdel-rahman Mohamed, George E. Dahl, Bhuvana Ramabhadran. 2015

502 citations recorded by the source corpus.

Convolutional Neural Networks (CNNs) are an alternative type of neural network that can be used to reduce spectral variations and model spectral correlations which exist in signals. Since speech signals exhibit both of these properties, we hypothesize that CNNs are a more effective model for speech compared to Deep Neural Networks (DNNs). In this paper, we explore applying CNNs to large vocabulary continuous speech recognition (LVCSR) tasks. First, we determine the appropriate architecture to make CNNs effective compared to DNNs for LVCSR tasks. Specifically, we focus on how many convolutional layers are needed, what is an appropriate number of hidden units, what is the best pooling strategy. Second, investigate how to incorporate speaker-adapted features, which cannot directly be modeled by CNNs as they do not obey locality in frequency, into the CNN framework. Third, given the importance of sequence training for speech tasks, we introduce a strategy to use ReLU+dropout during Hessian-free sequence training of CNNs. Experiments on 3 LVCSR tasks indicate that a CNN with the proposed speaker-adapted and ReLU+dropout ideas allow for a 12%-14% relative improvement in WER over a strong DNN system, achieving state-of-the art results in these 3 tasks.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

End-to-end Continuous Speech Recognition using Attention-based Recurrent NN: First Results (2014, background): “…has recently made important progress by adopting, as an accoustic model, deep neural networks such as fully-connected feedforward neural networks (A Mohamed et al. , 2009; Hinton et al. , 2012a), convolutional networks (Sainath et al. , 2014), and recurrent networks (RNN, Graves et al. , 2013a).”
Extracting Domain Invariant Features by Unsupervised Learning for Robust Automatic Speech Recognition (2018, background): “Recently, neural network-based acoustic models [1, 2, 3] have greatly improved the performance of automatic speech recognition (ASR) systems.”
Sequence-to-point learning with neural networks for nonintrusive load monitoring (2016, methodology): “We call this type of method a sequence-to-point learning method, which is widely applied for modelling the distributions of speech and image (Sainath et al. 2015; van den Oord et al. 2016; van den Oord, Kalchbrenner, and Kavukcuoglu 2016).”
Activation Functions: Comparison of trends in Practice and Research for Deep Learning (2018, background): “recognition [27], [28], segmentation [29], [30], scene understanding and description [31], [32], machine translation [33], [34] test to speech systems [35], [36], cancer detection systems [37], [38], [39], finger print detection [40], [41], weather forecast [42], [43], self-driving cars [44], [45], ”
Very deep convolutional neural networks for robust speech recognition (2016, methodology): “Based on the previous work using CNNs for ASR, the configuration described in [9] is usually used as the standard CNN architecture for speech recognition, with two convolutional layers followed by four fully connected layers.”
Phone recognition with hierarchical convolutional deep maxout networks (2015, methodology): “preferred to work with just 9 frames [14].”
Very Deep Convolutional Neural Networks for Noise Robust Speech Recognition (2016, methodology): “The baseline CNN utilizes the classical CNN configuration as in [14], with 2 convolutional layers each having 256 feature maps.”
Developing crossmodal expression recognition based on a deep neural model (2016, methodology): “…(1998) and used for different visual tasks, as demonstrated by the works of Lawrence, Giles, Tsoi, and Back (1997), Karnowski, Arel, and Rose (2010) and Khalil-Hani and Sung (2014), and auditory tasks in the works of Sainath et al. (2015), Li, Chan, and Chun (2010) and Schluter and Bock (2014).”
The NTT CHiME-3 system: Advances in speech enhancement and recognition for mobile multi-microphone devices (2015, methodology): “For conventional CNNs [19, 20], we experimented with two con ﬁ gurations: one with two convolution layers and one with three convolution layers.”
Wise teachers train better DNN acoustic models (2016, methodology): “Alternative, more complex architectures such as convolutional neural networks (CNN) offer promising gains over DNN, especially in joint architectures with DNN trained on SA features [11].”
A survey of deep neural network architectures and their applications (2017, methodology): “It should be noted that the CNNs can be regarded as an alternat iv model for speech recognition [134], [139].”
Building DNN acoustic models for large vocabulary speech recognition (2014, methodology): “Our formulation is consistent with other recent work on DCNN acoustic models [29], but we do not evaluate specialized feature post-processing or combining DNNs with DCNNs to form an ensemble of acoustic models.”
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition (2016, background): “Deep CNNs have had a major impact on fields, like content recommendation [24], speech recognition [25] and in computer vision [26–28], where they have become the de facto standard approach.”
Review of Image Classification Algorithms Based on Convolutional Neural Networks (2021, methodology): “However, the emergence of CNNs has made a series of breakthroughs in the field of image classification and has achieved excellent performance on large-scale visual tasks [19,24,77,81].”