Audio Super Resolution using Neural Networks

By Volodymyr Kuleshov, S. Zayd Enam, Stefano Ermon. 2017

160 citations recorded by the source corpus.

We introduce a new audio processing technique that increases the sampling rate of signals such as speech or music using deep convolutional neural networks. Our model is trained on pairs of low and high-quality audio examples; at test-time, it predicts missing samples within a low-resolution signal in an interpolation process similar to image super-resolution. Our method is simple and does not involve specialized audio processing techniques; in our experiments, it outperforms baselines on standard speech and music benchmarks at upscaling ratios of 2x, 4x, and 6x. The method has practical applications in telephony, compression, and text-to-speech generation; it demonstrates the effectiveness of feed-forward convolutional architectures on an audio generation task.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Real-Time Speech Frequency Bandwidth Extension (2020, methodology): “Inspired by the early success in image super-resolution [4], end-to-end audio-based solutions were proposed, based on wave-to-wave UNet [5], WaveNet [6, 7], hybrid time/frequency domain models [8].”
Upsampling Artifacts in Neural Audio Synthesis (2020, methodology): “, for speech enhancement [10], bandwidth extension [12] and voice conversion [13].”
Self-Supervised Generation of Spatial Audio for 360 Video (2018, methodology): “In the audio domain, self-supervision has also enabled the detection of sound-video misalignment [36] and audio super-resolution [30].”
Bandwidth Extension on Raw Audio via Generative Adversarial Networks (2019, background): “Table 1 shows that MU-GAN8 often performs worse in terms of SNR compared to the baseline models, but has lower LSD and higher MOS-LQO.”
Speech Super-Resolution Using Parallel WaveNet (2018, methodology): “Because super-resolution has little dependence on very far contexts, convolutional neural networks (CNNs) are frequently used in both image and audio super-resolution task [1, 2, 3].”
Time-Frequency Networks for Audio Super-Resolution (2018, methodology): “In [2], Kuleshov et al .”
Speech Bandwidth Extension with Wavenet (2019, methodology): “…of speech, using techniques like Gaussian mixture models and hidden Markov models [5]; more recently interest has grown in using neural networks either to model spectral envelopes [8] or to predict the upsampled waveform directly [6, 9, 10], leading to quality gains over earlier methods.”
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models (2023, methodology): “For super-resolution, we employ two models AudioUNet (Kuleshov et al., 2017) and NVSR (Liu et al., 2022a) as baseline models, and employ log-spectral distance (LSD) (Wang & Wang, 2021) as the evaluation metric for comparison.”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “Compared to previous work with 16kHz as the target frequency [5, 25, 39, 46, 53, 55, 56, 84], NU-Wave [49] is the first to synthesize 48kHz waveforms from 16kHz or 24kHz inputs, and also the first to apply diffusion models for audio super-resolution.”
NU-GAN: High resolution neural upsampling with GAN (2020, methodology): “[19] attempts to solve audio superresolution by applying highly expressive image to image translation algorithms [20] for audio translation, upsampling audio upto 8× with a maximum resolution of 16 kHz.”
Exploiting time-frequency patterns with LSTM-RNNs for low-bitrate audio restoration (2019, methodology): “We apply two metrics to objectively measuring music audio quality, which have commonly used for assessing the quality of enhanced signals [6, 15, 23, 29].”
Adversarial Training for Speech Super-Resolution (2019, methodology): “While some works focused on predicting the wide-band spectral envelope, others focused on directly estimating the missing data points [6], [7], [16], [17].”
NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling (2021, unknown): “There are several works reported that SNR is not effective in the upsampling task because it could not measure high-frequency generation [2, 5].”
HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, background): “Frequency bandwidth extension (Kuleshov et al., 2017; Lin et al., 2021) (also known as audio super-resolution) can be viewed as a realistic increase of signal sampling frequency.”