Analysis of CNN-based speech recognition system using raw speech as input

By Dimitri Palaz, Mathew Magimai-Doss, Ronan Collobert. 2015

291 citations recorded by the source corpus.

Abstract Automaticspeechrecognitionsystemstypicallymodeltherela-tionship between the acoustic speech signal and the phones intwo separate steps: feature extraction and classier training. Inourrecentworks, wehaveshownthat, intheframeworkofcon-volutionalneuralnetworks(CNN),therelationshipbetweentheraw speech signal and the phones can be directly modeled andASR systems competitive to standard approach can be built. Inthis paper, we rst analyze and show that, between the rst twoconvolutional layers, the CNN learns (in parts) and models thephone-specic spectral envelope information of 2-4 ms speech.Given that we show that the CNN-based approach yields ASRtrends similar to standard short-term spectral based ASR sys-tem under mismatched (noisy) conditions, with the CNN-basedapproach being more robust.Index Terms: automatic speech recognition, convolutionalneural networks, raw signal, robust speech recognition. 1. Introduction State-of-the-art automatic speech recognition (ASR) systemstypically model the relationship between the acoustic speechsignal and the phones in two separate steps, which are op-timized in an independent manner [1]. In a rst step, thespeech signal is transformed into features, usually composed ofa dimensionality reduction phase and an information selectionphase, based on the task-specic knowledge of the phenomena.These two phases have been carefully hand-crafted, leading tostate-of-the-art features such as Mel frequency cepstral coef-cients(MFCCs)orperceptuallinearpredictioncepstralfeatures(PLPs). In a second step, the likelihood of subword units suchas, phonemes is estimated using generative models or discrimi-native models.In recent years, in the hybrid HMM/ANN framework [1],there has been growing interests in using intermediate rep-resentations instead of conventional features, such as cepstral-based features, as input for neural networks-based systems.ANNs with deep learning architectures, more precisely, deepneural networks (DNNs) [2, 3], which can yield better systemthan a single hidden layer MLP have been proposed to addressvarious aspects of acoustic modeling. More specically, useof context-dependent phonemes [4, 5]; use of spectral featuresas opposed to cepstral features [6, 7]; CNN-based system withMel lter bank energies as input [8, 9, 10]; combination of dif-ferent features [11], to name a few. Features learning from therawspeechsignalusingneuralnetworks-basedsystemshasalsobeen investigated in [12]. In all these approaches, the features

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, background): “Similarly, when applied to audio waveforms, neural networks have been shown to learn auditory-like frequency decompositions on music [22] and speech [23], [24], [25], [26] in their lower layers.”
Wav2Letter: an End-to-End ConvNet-based Speech Recognition System (2016, background): “Other alternatives have been explored in the context of hybrid HMM/NN systems, such as the MMI criterion [2] which maximizes the mutual information between the acoustic sequence and word sequences or the Minimum Bayse Risk (MBR) criterion [5].”
Recent Trends in Deep Learning Based Natural Language Processing (2017, background): “[61] performed extensive analysis of CNN-based speech recognition systems when given raw speech as input.”
Sample-level Deep Convolutional Neural Networks for Music Auto-tagging Using Raw Waveforms (2017, methodology): “There are some works that use 0.6 ms (10 samples at 16 kHz audio) as a stride length [6, 7], but they used a CNN model only with three convolution layers, which is not sufﬁcient to learn the complex structure of musical signals.”
Acoustic Modelling from the Signal Domain Using CNNs (2016, methodology): “The filters learned can model the spectral envelope of speech signal corresponding to different phonemes [6].”
Speaker Recognition from Raw Waveform with SincNet (2018, background): “To mitigate this drawback, some recent works have proposed directly feeding the network with spectrogram bins [18–20] or even with raw waveforms [21–30].”
Interpretable Convolutional Filters with SincNet (2018, background): “This approach has shown promise in speech [25, 26, 27, 28, 29], including emotion tasks [30], speaker recognition [35], spoofing detection [34], and speech synthesis [31, 32].”
Recent trends in deep learning based personality detection (2019, background): “Palaz et al. (2015) analyze the CNN to understand the speech information that is modeled between the first two convolution layers.”
Learning Multiscale Features Directly from Waveforms (2016, background): “A popular approach is pass the waveform through strided convolutions, or networks connected to local temporal frames, often followed by a pooling step to create invariance to phase shifts and further downsample the signal [3, 4, 5, 6, 7, 8].”
Raw waveform-based speech enhancement by fully convolutional networks (2017, background): “Therefore, most studies have employed the CNN model for analyzing raw waveforms [22-26, 28].”
Residual Convolutional CTC Networks for Automatic Speech Recognition (2017, background): “Palaz et al. (2015) showed that CNN-based speech recognition which uses raw speech as input can be more robust.”
Deep Convolutional and LSTM Recurrent Neural Networks for Multimodal Wearable Activity Recognition (2016, background): “However, applying them to features derived from the raw sensor signals often leads to higher performance [21].”
Deep Convolutional Neural Networks with Layer-Wise Context Expansion and Attention (2016, methodology): “Various CNN structures [6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24] have been evaluated for speech recognition.”
Adieu features? End-to-end speech emotion recognition using a deep convolutional recurrent network (2016, methodology): “[10, 11] used CNNs directly trained on the speech signal to estimate phoneme class conditional probabilities and observed that the features learnt between the first two convolution layers tend to model the phone-specific spectral envelope of sub-segmental speech signal, which leads to a more robust ”
Recent Trends in Deep Learning Based Natural Language Processing [Review Article] (2018, background): “[60] performed extensive analysis of CNNbased speech recognition systems when given raw speech as input.”
A deep scattering spectrum — Deep Siamese network pipeline for unsupervised acoustic modeling (2016, background): “The untransformed waveform is the richest representation possible, and some supervised systems are able to learn phonetic classes directly from it [7].”
FeaStNet: Feature-Steered Graph Convolutions for 3D Shape Analysis (2017, methodology): “This has been demonstrated with for instance 1D convolution for audio signal [18] and 3D convolution over space and time for video signal [28].”