MelNet: A Generative Model for Audio in the Frequency Domain

By Sean Vasquez, Michael Lewis. 2019

142 citations recorded by the source corpus.

Capturing high-level structure in audio waveforms is challenging because a single second of audio spans tens of thousands of timesteps. While long-range dependencies are difficult to model directly in the time domain, we show that they can be more tractably modelled in two-dimensional time-frequency representations such as spectrograms. By leveraging this representational advantage, in conjunction with a highly expressive probabilistic model and a multiscale generation procedure, we design a model capable of generating high-fidelity audio samples which capture structure at timescales that time-domain models have yet to achieve. We apply our model to a variety of audio generation tasks, including unconditional speech generation, music generation, and text-to-speech synthesis---showing improvements over previous approaches in both density estimates and human judgments.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

Jukebox: A Generative Model for Music (2020, background): “While the approaches above serve as sophisticated generative models for raw audio to be conditioned on a compact and controllable representation of audio such as Mel spectrograms, Mel-Net (Vasquez & Lewis, 2019) (Razavi et al., 2019) do not suffer from this issue, and thus we use their approach.”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, background): “Note that, unconditional generation of audio in the frequency domain is considered easier – in the sense that the spectrogram is much shorter (e.g., 200 × ) than raw waveform (Vasquez & Lewis, 2019; Engel et al., 2019; Palkama et al., 2020).”
Guided-TTS: A Diffusion Model for Text-to-Speech via Classifier Guidance (2021, background): “Unconditional Speech Generation In general, the unconditional speech generative model (van den Oord et al., 2016; Vasquez & Lewis, 2019), which models audio without any information, is more challenging than the conditional generative model that synthesizes speech using text or melspectrograms.”
A Survey on Neural Speech Synthesis (2021, methodology): “, Flow, GAN, VAE, Diffusion) Besides the above acoustic models, there are a lot of other acoustic models [361, 22, 123, 183, 54], as shown in Table 2.”
RAVE: A variational autoencoder for fast and high-quality neural audio synthesis (2021, background): “Recent works in deep audio modelling have allowed novel types of interaction such as unconditional generation (Oord et al., 2016; Vasquez & Lewis, 2019; Dhariwal et al., 2020) or timbre transfer between instruments (Mor et al., 2018).”
High Fidelity Speech Synthesis with Adversarial Networks (2019, positive): “Tacotron (Wang et al., 2017) and MelNet (Vasquez & Lewis, 2019) constitute notable exceptions, and they use the Grifﬁn-Lim algorithm (Grifﬁn & Lim, 1984) to reconstruct missing phase information, which the models themselves do not generate.”
End-to-End Adversarial Text-to-Speech (2020, neutral): “Several mechanisms have been proposed to exploit monotonicity in tasks that require sequence alignment, including attention mechanisms [11, 20, 24, 48, 62, 72], loss functions [19, 21] and search-based approaches [29].”
Incremental Text to Speech for Neural Sequence-to-Sequence Models using Reinforcement Learning (2020, background): “However, these attention mechanisms generate hard attention weights which is undesirable for TTS [28].”
WaveNODE: A Continuous Normalizing Flow for Speech Synthesis (2020, background): “…consist of two stages: (1) transforming character embeddings to acoustic features such as mel-spectrograms (Ping et al., 2017; Shen et al., 2018; Vasquez & Lewis, 2019; Ren et al., 2019), and (2) synthesizing time-domain waveforms from the derived acoustic features (Oord et al., 2016; 2017;…”
Semi-Supervised Generative Modeling for Controllable Speech Synthesis (2019, methodology): “…factors between these models include the degree of parallelism, with some models using Transformer based architectures (Ren et al., 2019), the choice of conditional independence assumptions made (Vasquez & Lewis, 2019) or the number of separately trained components (Gib-iansky et al., 2017).”
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning (2020, methodology): “Neural Autoregressive Density Estimator (NADE) [34] is a prominent earlier work followed by more recent Wavenet [35], SampleRNN [36] and MelNet [37].”
Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “Vasquez S, Lewis M (2019) Melnet: A generative model for audio in the frequency domain. arXiv preprint arXiv:190601083 222.”
Revisiting Over-Smoothness in Text to Speech (2022, background): “Consequently, the melspectrograms following dependent and multimodal distributions cannot be well modeled by the MAE or MSE loss, which presents great challenges in non-autoregressive TTS modeling and causes over-smoothed (blurred) predictions in melspectrograms (Vasquez and Lewis, 2019; Sheng and P”
High Quality Audio Coding with Mdctnet (2022, background): “Similar ideas have been considered for speech coding [21], audio synthesis [22], and music source separation [23].”
Speaker Conditional WaveRNN: Towards Universal Neural Vocoder for Unseen Speaker and Recording Conditions (2020, result): “Other recent works have been also found in literature, notable among them are SampleRNN [20], WaveGlow [21], LPCNet [22] and MelNet [23].”
A Survey on Audio Diffusion Models: Text To Speech Synthesis and Enhancement in Generative AI (2023, unknown): “In essence, an audio waveform is a mixture of frequencies [14, 120], and therefore, audio analysis often starts with transforming the audio from its raw waveform in the time domain to the spectrogram in the time-frequency domain.”
High Quality Streaming Speech Synthesis with Low, Sentence-Length-Independent Latency (2020, methodology): “Our model uses a variation of GMM attention, similar to [21], where the mixture of Gaussian distributions is replaced by a Mixture of Logistic distributions (MoL), hence we refer to it as MoL attention.”