NU-Wave: A Diffusion Probabilistic Model for Neural Audio Upsampling

By Junhyeok Lee, Seungu Han. 2021

102 citations recorded by the source corpus.

In this work, we introduce NU-Wave, the first neural audio upsampling model to produce waveforms of sampling rate 48kHz from coarse 16kHz or 24kHz inputs, while prior works could generate only up to 16kHz. NU-Wave is the first diffusion probabilistic model for audio super-resolution which is engineered based on neural vocoders. NU-Wave generates high-quality audio that achieves high performance in terms of signal-to-noise ratio (SNR), log-spectral distance (LSD), and accuracy of the ABX test. In all cases, NU-Wave outperforms the baseline models despite the substantially smaller model capacity (3.0M parameters) than baselines (5.4-21%). The audio samples of our model are available at https://mindslab-ai.github.io/nuwave, and the code will be made available soon.

Subfields: Biomedical Engineering, Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Biomedical Engineering

1. Jordi Pons
2. Sungroh Yoon
3. Xu Tan

In Computer Vision and Pattern Recognition

1. Tie-Yan Liu
2. Tao Qin
3. Xu Tan
4. Jordi Pons
5. Joan Serrà
6. Chaehun Shin
7. Wei Ping
8. Santiago Pascual

In Speech & Audio

1. Xu Tan
2. Tao Qin
3. Tie-Yan Liu
4. Heeseung Kim
5. Wei Ping
6. Sungroh Yoon
7. Jordi Pons
8. Sheng Zhao

In Signal Processing

1. Xu Tan
2. Tie-Yan Liu
3. Tao Qin
4. Wei Ping
5. Sungroh Yoon
6. Heeseung Kim
7. Jordi Pons
8. Santiago Pascual

Citations in the corpus

• On Fast Sampling of Diffusion Probabilistic Models (2021, background): “, 2021), image or audio super-resolution (Li et al., 2021; Lee and Han, 2021), text-to-speech (Jeong et al.”
• PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Driven Adaptive Prior (2021, background): “Speech synthesis is one of the most successful applications, where the diffusion model can synthesize spectral or time-domain audio conditioned on text or spectral information, respectively, achieving a competitive quality but faster sampling [2, 16, 10, 18] than autoregressive models [25, 11].”
• A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “NU-Wave [49] empirically finds that the receptive field for a conditional signal should be larger than that of the noisy input, and thus modifies the local conditioner for the information ensemble of signals with different receptive fields and upscaling ratios.”
• PriorGrad: Improving Conditional Denoising Diffusion Models with Data-Dependent Adaptive Prior (2021, background): “…spectral or time-domain audio conditioned on text or spectral information, respectively, achieving a competitive quality but faster sampling (Chen et al., 2021; Kong et al., 2021; Jeong et al., 2021; Lee & Han, 2021) than autoregressive models (Oord et al., 2016b; Kalchbrenner et al., 2018).”
• Universal Speech Enhancement with Score-based Diffusion (2022, methodology): “We next consider an approach inspired by NU-Wave (Lee & Han, 2021).”