Upsampling Artifacts in Neural Audio Synthesis

By Jordi Pons, Santiago Pascual, Giulio Cengarle, Joan Serrà. 2020

77 citations recorded by the source corpus.

A number of recent advances in neural audio synthesis rely on up-sampling layers, which can introduce undesired artifacts. In computer vision, upsampling artifacts have been studied and are known as checkerboard artifacts (due to their characteristic visual pattern). However, their effect has been overlooked so far in audio processing. Here, we address this gap by studying this problem from the audio signal processing perspective. We first show that the main sources of upsampling artifacts are: (i) the tonal and filtering artifacts introduced by problematic upsampling operators, and (ii) the spectral replicas that emerge while upsampling. We then compare different upsampling layers, showing that nearest neighbor upsamplers can be an alternative to the problematic (but state-of-the-art) transposed and subpixel convolutions which are prone to introduce tonal artifacts.

Subfields: Cognitive Neuroscience, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Cognitive Neuroscience

In Signal Processing

Citations in the corpus

High-Fidelity Audio Compression with Improved RVQGAN (2023, background): “The upsampling layers of the decoder introduce significant aliasing artifacts [29].”
GAN Vocoder: Multi-Resolution Discriminator Is All You Need (2021, methodology): “We use nearest-neighbor upsampling over transposed convolution as [24, 25] report various advantages (e.”
BigVGAN: A Universal Neural Vocoder with Large-Scale Training (2022, background): “We also tried out nearest-neighbor upsampling which is reported to have less artifacts for audio synthesis (Pons et al., 2021), but it also resulted in the early collapse.”
End-to-End Neural Audio Coding in the MDCT Domain (2023, background): “In addition, up-sampling artifacts [19] remain as a potential threat to improving the perceptual quality of reconstruction audio signals decoded with CNNs. Based on these ideas, we propose a new frequency-domain neural audio coding framework that can be optimized in an end-to-end manner with a…”
Fre-GAN: Adversarial Frequency-consistent Audio Synthesis (2021, methodology): “To upsample lower scale waveforms, we use Nearest Neighbor (NN) upsampler [18] which has been proven to alleviate tonal artifacts caused by transposed convolutions [19].”
Avocodo: Generative Adversarial Network for Artifact-free Vocoder (2022, methodology): “However, up-sampling layer, such as transposed convolution, causes several artifacts, tonal artifacts (Pons et al. 2021) being one example.”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “Compared to previous work with 16kHz as the target frequency [5, 25, 39, 46, 53, 55, 56, 84], NU-Wave [49] is the first to synthesize 48kHz waveforms”
Universal Speech Enhancement with Score-based Diffusion (2022, methodology): “If the convolutional block will be used for up-sampling, the core’s input is processed initially by a transposed convolutional layer of kernel width and stride equal to the up-sampling ratio, without padding (Pons et al., 2021).”