Learning Multiscale Features Directly from Waveforms

By Zhenyao Zhu, Jesse Engel, Awni Y. Hannun. 2016

66 citations recorded by the source corpus.

Deep learning has dramatically improved the performance of speech recognition systems through learning hierarchies of features optimized for the task at hand. However, true end-to-end learning, where features are learned directly from waveforms, has only recently reached the performance of hand-tailored representations based on the Fourier transform. In this paper, we detail an approach to use convolutional filters to push past the inherent tradeoff of temporal and frequency resolution that exists for spectral representations. At increased computational cost, we show that increasing temporal resolution via reduced stride and increasing frequency resolution via additional filters delivers significant performance improvements. Further, we find more efficient representations by simultaneously learning at multiple scales, leading to an overall decrease in word error rate on a difficult internal speech test set by 20.7% relative to networks with the same number of parameters trained on spectrograms.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

GANSynth: Adversarial Neural Audio Synthesis (2019, background): “Convolu-tional ﬁlters trained for different tasks on this data commonly learn to form logarithmically-scaled frequency selective ﬁlter banks spanning the range of human hearing (Dieleman & Schrauwen, 2014; Zhu et al., 2016).”
Temporal Segment Networks for Action Recognition in Videos (2017, methodology): “The submission is an ensemble of TSN models trained on training and validation data with Inception V3 and ResNet-152 architectures, and the audio models [76] trained on audio signals of the videos.”
End-to-End Environmental Sound Classification using a 1D Convolutional Neural Network (2019, background): “Zhu et al. (2016) proposed an end-to-end learning approach for speech recognition based on multiscale convolutions that learns the representation directly from audio waveforms.”
Wavelet Networks: Scale Equivariant Learning From Raw Waveforms (2020, background): “Though approaches relying on layer-wise multi-scale representations exist [84, 43, 72, 25, 83], they are unable to preserve the same sense of scale across layers as they do not commute with scaling.”
On the limit of English conversational speech recognition (2021, background): “Studies on acoustic modeling of raw waveforms, however, suggest that higher temporal resolution might be beneficial [22, 23].”