Learning a better representation of speech soundwaves using restricted boltzmann machines

By Navdeep Jaitly, Geoffrey E. Hinton. 2011

223 citations recorded by the source corpus.

State of the art speech recognition systems rely on preprocessed speech features such as Mel cepstrum or linear predictive coding coefficients that collapse high dimensional speech sound waves into low dimensional encodings. While these have been successfully applied in speech recognition systems, such low dimensional encodings may lose some relevant information and express other information in a way that makes it difficult to use for discrimination. Higher dimensional encodings could both improve performance in recognition tasks, and also be applied to speech synthesis by better modeling the statistical structure of the sound waves. In this paper we present a novel approach for modeling speech sound waves using a Restricted Boltzmann machine (RBM) with a novel type of hidden variable and we report initial results demonstrating phoneme recognition performance better than the current state-of-the-art for methods based on Mel cepstrum coefficients.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “to learn auditory-like frequency decompositions on music [23] and speech [24]–[27] in their lower layers.”
A tutorial survey of architectures, algorithms, and applications for deep learning (2014, background): “The work also showed the importance of raw speech features of spectrogram --- back from the longpopular MFCC features, but not yet reaching the raw speechwaveform level (Sheikhzadeh and Deng, 1994; Jaitly and Hinton, 2011).”
Analysis of CNN-based speech recognition system using raw speech as input (2015, background): “Features learning from the raw speech signal using neural networks-based systems has also been investigated in [12].”
Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network (2016, unknown): “In one of the first studies that suggested learning better features for automatic speech recognition (ASR) that used directly the speech waveform was Jaitly and Hinton [6].”
Learning Multiscale Features Directly from Waveforms (2016, background): “A popular approach is pass the waveform through strided convolutions, or networks connected to local temporal frames, often followed by a pooling step to create invariance to phase shifts and further downsample the signal [3, 4, 5, 6, 7, 8].”
Towards deeper understanding: Deep convex networks for semantic utterance classification (2012, methodology): “Following the introduction of this RBM learning and layer-by-layer construction of deep architectures, DBNs have been successfully used for numerous tasks in speech and language processing, including acoustic modeling [8, 9, 10] and intent determination [6].”
A review of unsupervised feature learning and deep learning for time-series modeling (2014, background): “Only a few works have attempted this with speech recognition (Jaitly and Hinton, 2011; Smith and Lewicki, 2005) and physiological data (Wulsin et al.”
Learning Invariant Representations of Molecules for Atomization Energy Prediction (2012, background): “…1This approach has already been applied in multiple domains such as natural language processing (Collobert et al., 2011) or speech recognition (Jaitly and Hinton, 2011). inherent problem of the Coulomb matrix descriptor is that it lacks invariance with respect to permutation of atom…”
Towards End-To-End Speech Recognition with Recurrent Neural Networks (2014, background): “Although it is possible to directly transcribe raw speech waveforms with RNNs (Graves, 2012, Chapter 9) or features learned with restricted Boltzmann machines (Jaitly & Hinton, 2011), the computational cost is high and performance tends to be worse than conventional preprocessing.”
A Deep Neural Network for Acoustic-Articulatory Speech Inversion (2011, methodology): “Deep architectures have recently been used to obtain state-of-the-art accuracies in phone classification [9, 10, 11, 12] (i.”
Recent progresses in deep learning based acoustic models (2017, methodology): “Inspired by the end-to-end processing in the machine learning community, there are always efforts [51]–[54] trying to replace the Mel-ﬁlter-bank extraction by directly learning ﬁlters using a network to process the raw speech waveforms and training it jointly with the recognizer network.”
End-to-end Phoneme Sequence Recognition using Convolutional Neural Networks (2013, background): “The work presented in [19] successfully investigates features learning from raw speech for phoneme recognition.”
Multichannel Signal Processing With Deep Neural Networks for Automatic Speech Recognition (2017, background): “The mic array is placed at the center of the room and closer to the wall, respectively, to capture reasonably different reverberation characteristics.”
Learning the speech front-end with raw waveform CLDNNs (2015, methodology): “Past work has achieved phase invariance using convolutional layers which pool in time [4, 5, 7] or DNN layers with large, potentially overcomplete, hidden units [6], which can capture the same filter shape at a variety of phases.”
Deep Convex Net: A Scalable Architecture for Speech Pattern Classification (2011, methodology): “And many years later the use of more powerful Restricted Boltzmann Machine (RBM) overcomes some difficulty encountered earlier [10].”