Feature Learning in Deep Neural Networks - A Study on Speech Recognition Tasks

By Dong Yu, M. Seltzer, Jinyu Li, Jui-Ting Huang, Frank Torsten Bernd Seide. 2013

266 citations recorded by the source corpus.

Recent studies have shown that deep neural networks (DNNs) perform significantly better than shallow networks and Gaussian mixture models (GMMs) on large vocabulary speech recognition tasks. In this paper, we argue that the improved accuracy achieved by the DNNs is the result of their ability to extract discriminative internal representations that are robust to the many sources of variability in speech signals. We show that these representations become increasingly insensitive to small perturbations in the input with increasing network depth, which leads to better speech recognition performance with deeper networks. We also show that DNNs cannot extrapolate to test samples that are substantially different from the training examples. If the training data are sufficiently representative, however, internal features learned by the DNN are relatively stable with respect to speaker differences, bandwidth differences, and environment distortion. This enables DNN-based recognizers to perform as well or better than state-of-the-art systems based on GMMs or shallow networks without the need for explicit model adaptation or feature normalization.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Training Very Deep Networks (2015, positive): “Other results on practical machine learning problems have also underscored the superiority of deeper networks [6] in terms of accuracy and/or performance.”
Toward Domain-Invariant Speech Recognition via Large Scale Training (2018, methodology): “Similarly, mixed bandwidth training has been shown to handle multiple sample rates simultaneously, without any need for explicit reconstruction of missing bands [5].”
Unsupervised Learning of Disentangled and Interpretable Representations from Sequential Data (2017, background): “Learning to disentangle the mixed information from the surface representation is essential for a wide variety of speech applications: for example, noise robust speech recognition [41, 38, 37, 16], speaker verification [5], and voice conversion [40, 29, 24].”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, background): “For instance, previous work evaluated neural network acoustic models on phoneme recognition using different acoustic features [11] or investigated how such models learn invariant representations [12] and encode linguistic features [13, 14].”
Speech Emotion Recognition Using Deep Learning Techniques: A Review (2019, methodology): “The main problem that affects the overall performance of the RNN is its sensitivity towards the disappearance of gradients [84], [85].”
Recognizing Long-Form Speech Using Streaming End-to-End Models (2019, methodology): “Additionally, we generate both 8 kHz and 16 kHz versions of the training data to reduce acoustic mismatch due to sampling rates [19, 34].”
Speaker Adaptive Training of Deep Neural Network Acoustic Models Using I-Vectors (2015, background): “As with SAT of GMMs, SAT of DNNs starts from a SI-DNN model (on the right of Fig.”
Speech emotion recognition using deep neural network and extreme learning machine (2014, background): “A very promising characteristic of DNNs is that they can learn high-level invariant features from raw data [15, 4], which is potentially helpful for emotion recognition.”
Invariant Representations for Noisy Speech Recognition (2016, methodology): “The inﬂuence of representations produced by a neural network to internal noise reduction is discussed in Yu et al. (2013) and this work sets a baseline for experiments on Aurora-4 dataset.”
State-of-the-Art Deep Learning: Evolving Machine Intelligence Toward Tomorrow’s Intelligent Network Traffic Control Systems (2017, background): “The recent research in deep learning architectures and algorithms have revived the interest in neural networks and representation learning, particularly in the speech recognition area [71]–[73].”
On the Role of Nonlinear Transformations in Deep Neural Network Acoustic Models (2016, background): “Despite these advances, state-of-the-art ASR systems cannot match human-level performance in a variety of speech recognition tasks [6], motivating a number of recent studies aimed at better understanding deep learning in the hope of gaining intuitions that may lead to improved models [6, 7, 8, 9, 11”
Deep Neural Networks for Single-Channel Multi-Talker Speech Recognition (2015, methodology): “In these methods, DNNs are usually trained on clean and noisy stereo pairs of observations to minimize the mean squared error (MSE) between the estimated clean speech and the actual clean speech [17].”
Rectifier Nonlinearities Improve Neural Network Acoustic Models (2013, result): “The performance gains observed in our sigmoidal DNNs relative to the GMM baseline system are on par with other recent work with DNN acoustic models on the Switchboard task (Yu et al., 2013).”
Speech Emotion Recognition from Spectrograms with Deep Convolutional Neural Network (2017, methodology): “These methods have shown promising results in the fields of speech recognition, emotion recognition, and other speech analysis applications [14-16].”
On speaker adaptation of long short-term memory recurrent neural networks (2015, background): “Another issue that acoustic models, both GMMs and DNNs, encounter is the mismatch between acoustic models and testing speakers.”
An Overview of Noise-Robust Automatic Speech Recognition (2014, background): “With deep and wide hidden layers, the DNN provides a very strong normalization to heterogeneous data [77], [78], [287].”