Fast Spectrogram Inversion Using Multi-Head Convolutional Neural Networks

By Sercan Ö. Arik, Heewoo Jun, G. Diamos. 2018

120 citations recorded by the source corpus.

We propose the multi-head convolutional neural network (MCNN) for waveform synthesis from spectrograms. Nonlinear interpolation in MCNN is employed with transposed convolution layers in parallel heads. MCNN enables significantly better utilization of modern multi-core processors than commonly used iterative algorithms like Griffin–Lim, and yields very fast (more than 300 × real time) runtime. For training of MCNN, we use a large-scale speech recognition dataset and losses defined on waveforms that are related to perceptual audio quality. We demonstrate that MCNN constitutes a very promising approach for high-quality speech synthesis, without any iterative algorithms or autoregression in computations.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

Waveglow: A Flow-based Generative Network for Speech Synthesis (2018, methodology): “All three require compound loss functions to improve audio quality or problems with mode collapse [9, 7, 8].”
A Survey on Neural Speech Synthesis (2021, background): “Except for the regular GAN losses such as WGAN-GP [94], hinge-loss GAN [194], and LS-GAN [225], other specific losses such as STFT loss [10, 395] and feature matching loss [178] are also leveraged.”
DDSP: Differentiable Digital Signal Processing (2020, methodology): “For example, strided convolution models–such as SING (Defossez et al., 2018), MCNN (Arik et al., 2019), and WaveGAN (Donahue et al., 2019)–generate waveforms directly with overlapping frames.”
Parallel Wavegan: A Fast Waveform Generation Model Based on Generative Adversarial Networks with Multi-Resolution Spectrogram (2019, background): “, G(z)), and Lsc and Lmag denote spectral convergence and log STFT magnitude loss, respectively, which are defined as follows [23]:”
End-to-End Adversarial Text-to-Speech (2020, methodology): “l and the vocoder have to be trained sequentially. 6 Spectrogram prediction losses have been used extensively for feed-forward audio prediction models (Yamamoto et al., 2019; 2020; Yang et al., 2020; Arık et al., 2018; Engel et al., 2020; Wang et al., 2019). We note that the L 1 loss we use is compa”
MelNet: A Generative Model for Audio in the Frequency Domain (2019, background): “Several works have demonstrated that time-domain models can be used to invert spectral representations to high-fidelity audio [42, 38, 3], suggesting that MelNet could be used in concert with time-domain models such as WaveNet.”
Probability density distillation with generative adversarial networks for high-quality parallel waveform generation (2019, methodology): “where x and x̂ denote the target and the estimated speech signal; λMAG denotes a weight coefficient to balance two losses, spectral convergence (LSC) and log STFT magnitude loss (LMAG), which is defined as follows [28]:”
Multi-Band Melgan: Faster Waveform Generation For High-Quality Text-To-Speech (2020, methodology): “To solve this problem, we adopt the multi-resolution STFT loss, which has been proven to be more effective to measure the difference between fake and real speech [21, 23, 24].”
Wave-Tacotron: Spectrogram-Free End-to-End Text-to-Speech Synthesis (2020, methodology): “Other approaches have eschewed probabilistic models and used GANs [10, 11], or carefully constructed spectral losses [12, 13, 14].”
Robust universal neural vocoding (2018, background): “However, due to sparsity (it is unlikely that we will ever be able to cover all possible human-generated sounds) the neural vocoder models are prone to over-fit to the training speaker characteristics and have poor generalization capabilities [14].”
Parallel Neural Text-to-Speech (2019, methodology): “We define the STFT loss as the summation of `2 loss on the magnitudes of STFT and `1 loss on the log-magnitudes of STFT between the output audio and ground truth audio (Arık et al., 2019; Wang et al., 2019).”
Generative Adversarial Network based Speaker Adaptation for High Fidelity WaveNet Vocoder (2018, methodology): “Different from [6], the mean square loss of log-scale STFT-magnitude (log-mag loss) [15] were adopted instead of the power-loss.”
Towards Achieving Robust Universal Neural Vocoding (2018, unknown): “characteristics and have poor generalization capabilities [17].”
Universal Neural Vocoding with Parallel Wavenet (2021, background): “to overfitting to the training data, and do not generalise well to unseen voices [12].”
Unsupervised Cross-Domain Singing Voice Conversion (2020, methodology): “To mitigate this, we include a spectral amplitude distance loss [12, 39] in multiple FFT resolutions [3, 13, 14].”
Non-Autoregressive Neural Text-to-Speech (2019, methodology): “We define the STFT loss as the summation of `2 loss on the magnitudes of STFT and `1 loss on the log-magnitudes of STFT between the output audio and ground truth audio (Ping et al., 2018a; Arık et al., 2019; Wang et al., 2019).”
Improved Parallel Wavegan Vocoder with Perceptually Weighted Spectrogram Loss (2021, methodology): “[16] first proposed spectral convergence and log-scale STFT magnitude losses, and our previous work proposed combing these in a multi-resolution form [9].”
WG-WaveNet: Real-Time High-Fidelity Speech Synthesis without GPU (2020, background): “(1) where M is the number of different parameter sets of STFT; Lsc and Lmag are the spectral convergence loss and the log STFTmagnitude loss from [21]:”
Variable Bitrate Discrete Neural Representations via Causal Self-Attention (2021, methodology): “Following Arik et al. [1], we compute the following losses between the reference audio waveform x and reconstructed waveform ˆ x : To capture how the reconstructed waveform ﬁts large-amplitude spectral components, we compute Spectral convergence (SC) : where STFT is the Short-Time Fourier Transform…”