Estimating phoneme class conditional probabilities from raw speech signal using convolutional neural networks

By Dimitri Palaz, Ronan Collobert, Mathew Magimai-Doss. 2013

209 citations recorded by the source corpus.

In hybrid hidden Markov model/artificial neural networks (HMM/ANN) automatic speech recognition (ASR) system, the phoneme class conditional probabilities are estimated by first extracting acoustic features from the speech signal based on prior knowledge such as, speech perception or/and speech production knowledge, and, then modeling the acoustic features with an ANN. Recent advances in machine learning techniques, more specifically in the field of image processing and text processing, have shown that such divide and conquer strategy (i.e., separating feature extraction and modeling steps) may not be necessary. Motivated from these studies, in the framework of convolutional neural networks (CNNs), this paper investigates a novel approach, where the input to the ANN is raw speech signal and the output is phoneme class conditional probability estimates. On TIMIT phoneme recognition task, we study different ANN architectures to show the benefit of CNNs and compare the proposed approach against conventional approach where, spectral-based feature MFCC is extracted and modeled by a multilayer perceptron. Our studies show that the proposed approach can yield comparable or better phoneme recognition performance when compared to the conventional approach. It indicates that CNNs can learn features relevant for phoneme classification automatically from the raw speech signal.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

WaveNet: A Generative Model for Raw Audio (2016, methodology): “Traditionally, speech recognition research has largely focused on using log mel-filterbank energies or mel-frequency cepstral coefficients (MFCCs), but has been moving to raw audio recently (Palaz et al., 2013; Tüske et al., 2014; Hoshen et al., 2015; Sainath et al., 2015).”
Learning Filterbanks from Raw Speech for Phone Recognition (2017, background): “There have been several attempts at learning directly from the raw waveform for speech recognition [5, 6, 7, 8].”
Wav2Letter: an End-to-End ConvNet-based Speech Recognition System (2016, methodology): “…sequence with Tx frames of dx dimensional vectors, a convolution with kernel width kw, stride dw and dy frame size output computes the following: yit = bi + dx∑ j=1 kw∑ k=1 wi,j,k x j dw×(t−1)+k ∀1 ≤ i ≤ dy, (1) where b ∈ Rdy and w ∈ Rdy×dx×kw are the parameters of the convolution (to be learned).”
Recent advances in convolutional neural networks (2015, methodology): “ speech recognition errors by applying CNN on Mel lter bank features. Some attempts use the raw waveform with CNNs, and to learn lters to process the raw waveform jointly with the rest of the network [291, 292]. Most of the early applications of CNN in ASR only use fewer convolution layers. For exam”
Analysis of CNN-based speech recognition system using raw speech as input (2015, background): “We showed that these filters respond to different frequency bandwidths [16], and that they show some level of invariance across databases [17].”
Speech acoustic modeling from raw multichannel waveforms (2015, methodology): “This result extends the results in [2, 3] to multicondition training with noise and reverberation.”
Deep Scattering Spectrum with deep neural networks (2014, unknown): “For example, Palaz et al [2] showed that learning filters from a raw-signal jointly within a neural network framework was slightly worse than feeding log-mel spectra as input into the network.”
Multichannel Signal Processing With Deep Neural Networks for Automatic Speech Recognition (2017, background): “The mic array is placed at the center of the room and closer to the wall, respectively, to capture reasonably different reverberation characteristics.”
Learning the speech front-end with raw waveform CLDNNs (2015, methodology): “Past work has achieved phase invariance using convolutional layers which pool in time [4, 5, 7] or DNN layers with large, potentially overcomplete, hidden units [6], which can capture the same filter shape at a variety of phases.”
Convolutional Neural Networks-based continuous speech recognition using raw speech signal (2015, methodology): “The filter stage of the CNN-based system can be seen as a feature extractor or matching filters [5].”
Acoustic modeling with deep neural networks using raw time signal for LVCSR (2014, background): “Previously, a dramatic degradation of recognition accuracy by training a DNN directly on raw speech signal was reported in [9].”
Automatic Sleep Stage Scoring with Single-Channel EEG Using Convolutional Neural Networks (2016, methodology): “Again, there are characteristic examples from speech and acoustics in [6, 19, 27, 20, 10].”
End-to-end Phoneme Sequence Recognition using Convolutional Neural Networks (2013, background): “Finally, convolutional neural networks have shown the capability of learning features from raw speech and estimate phoneme conditional probabilities in a single system [20].”
Time-series modeling with undecimated fully convolutional neural networks (2015, methodology): “For example, in [6,7] a CNN was used to predict the phoneme class conditional probability for a segment of raw input signal.”
Kaleidoscope: An Efficient, Learnable Representation For All Structured Linear Maps (2020, background): “Other approaches that can learn from raw audio can be found in (Palaz et al., 2013; Collobert et al., 2016; Ghahremani et al., 2016).”
Convolutional Neural Networks for Distant Speech Recognition (2014, background): “Although limited weight sharing was reported to bring improvements for phone classification [25] and small LVSR tasks [32], a recent study on larger tasks [27] suggests that full weight sharing with a sufficient number of filters can work equally well, while being easier to implement.”
Advances in Very Deep Convolutional Neural Networks for LVCSR (2016, methodology): “CNNs with strided pooling in time have been succesfully used for small-footprint keyword spotting [25], learning filterbanks from raw signal [26], and the endto-end CTC-based model in [27].”