Very deep convolutional neural networks for raw waveforms

By Wei Dai, Chia Dai, Shuhui Qu, Juncheng Billy Li, Samarjit Das. IEEE International Conference on Acoustics, Speech, and Signal Processing, 2016

392 citations recorded by the source corpus.

Learning acoustic models directly from the raw waveform data with minimal processing is challenging. Current waveform-based models have generally used very few (∼2) convolutional layers, which might be insufficient for building high-level discriminative features. In this work, we propose very deep convolutional neural networks (CNNs) that directly use time-domain waveforms as inputs. Our CNNs, with up to 34 weight layers, are efficient to optimize over very long sequences (e.g., vector of size 32000), necessary for processing acoustic waveforms. This is achieved through batch normalization, residual learning, and a careful design of down-sampling in the initial layers. Our networks are fully convolutional, without the use of fully connected layers and dropout, to maximize representation learning. We use a large receptive field in the first convolutional layer to mimic bandpass filters, but very small receptive fields subsequently to control the model capacity. We demonstrate the performance gains with the deeper models. Our evaluation shows that the CNN with 18 weight layers outperforms the CNN with 3 weight layers by over 15% in absolute accuracy for an environmental sound recognition task and is competitive with the performance of models using log-mel features.

Subfields: Artificial Intelligence, Signal Processing

View Paper: https://www.semanticscholar.org/paper/cddd943dae3a6ca5c591d0c5be0bb5b490b28c63

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

A Convolutional Neural Network to Filter Artifacts in Spectroscopic Magnetic Resonance Imaging (2018, background): “18 Deep convolutional neural networks (CNNs) in particular are well suited to analyzing waveforms similar to raw spectra, 19,20 and have only recently been applied in the context of MRS. 21 Convolutional neural networks consist of sequential layers of convolution, down-sampling, and logistic…”
PANNs: Large-Scale Pretrained Audio Neural Networks for Audio Pattern Recognition (2019, methodology): “1) DaiNet: DaiNet [33] applied a kernel with a length of 80 and a stride of 4 to the input waveform of audio recordings.”
Compact and Computationally Efficient Representation of Deep Neural Networks (2018, neutral): “…Ω :[0 , 4 , 3 , 2] colI :[4 , 9 , 11 , 1 , 8 , 3 , 7 , 0 , 1 , 5 , 8 , 9 , 11 , 0 , 3 , 7 , 2 , 9 , 3 , 4 , 5 , 8 , 9 , 7 , 1 , 2 , 5 , 7] Ω Ptr :[0 , 3 , 5 , 7 , 13 , 16 , 17 , 18 , 23 , 24 , 28] rowPtr :[0 , 3 , 4 , 7 , 9 , 10] Notice that we can uniquely reconstruct M from this data structure.”
CNN-Based Learnable Gammatone Filterbank and Equal-Loudness Normalization for Environmental Sound Classification (2020, background): “Su-pervised [15], [16], [18]–[21] and unsupervised [17] approaches have been proposed for ESC.”
Analyzing Learned Representations of a Deep ASR Performance Prediction Model (2018, methodology): “For speech input, we use the best architecture (m18) proposed in (Dai et al., 2017) (colored in red in Figure 1) of dimensions 48000 x 1 (48000 samples correspond to 6s of speech).”
Connecting the Dots between Audio and Text without Parallel Data through Visual Knowledge Transfer (2021, background): “Some prior work in learning sound event representations are supervised by category labels (Dai et al., 2017; Boddapati et al., 2017; Kumar et al., 2018; Guzhov et al., 2021b; Gong et al., 2021).”
Between-Class Learning for Image Classification (2017, background): “In fact, recent studies have demonstrated that CNNs can learn speech and sounds directly from raw waveforms, and each filter learns to respond to a particular frequency area [21, 26, 7].”
Wavelet Networks: Scale Equivariant Learning From Raw Waveforms (2020, methodology): “[16], and the MTAT experiments of Lee et al.”
End-to-End Environmental Sound Classification using a 1D Convolutional Neural Network (2019, methodology): “Dai et al. (2017) proposed several very deep convolutional models for environmental sound classiﬁcation that achieved 72% of accuracy on UrbanSound8k dataset.”
Realistic Speech-Driven Facial Animation with GANs (2019, background): “The initial convolutional layer starts with a large kernel, as recommended in [14], which helps limit the depth of the network while ensuring that the low-level features are meaningful.”
Raw waveform-based speech enhancement by fully convolutional networks (2017, background): “Therefore, most studies have employed the CNN model for analyzing raw waveforms [22-26, 28].”
SpeechMix - Augmenting Deep Sound Recognition Using Hidden Space Interpolations (2020, background): “M18 [26] is a very deep CNN that directly uses time-domain wave-forms as input.”
Earthquake transformer—an attentive deep-learning model for simultaneous earthquake detection and phase picking (2020, background): “Several studies 36 – 38 have shown in end-to-end learning from raw waveforms that employing deeper networks can be bene ﬁ cial for having more expressive power, better generalization, and more robustness to noise in the waveform.”
An Attention-Based Deep Learning Approach for Sleep Stage Classification With Single-Channel EEG (2021, methodology): “This is inspired by some previous works that used multiple CNN kernel sizes to extract different frequency features ( i.e. , low-and high-frequencies) such as [27], [28].”
Interpreting and Explaining Deep Neural Networks for Classification of Audio Signals (2018, methodology): “Audio classiﬁcation is often based on spectrogram representations of the data [26] but successful classiﬁcation based on raw waveform data has been reported as well [17].”