DDSP: Differentiable Digital Signal Processing

By Jesse Engel, Lamtharn Hantrakul, Chenjie Gu, Adam Roberts. 2020

489 citations recorded by the source corpus.

Most generative models of audio directly generate samples in one of two domains: time or frequency. While sufficient to express any signal, these representations are inefficient, as they do not utilize existing knowledge of how sound is generated and perceived. A third approach (vocoders/synthesizers) successfully incorporates strong domain knowledge of signal processing and perception, but has been less actively researched due to limited expressivity and difficulty integrating with modern auto-differentiation-based machine learning methods. In this paper, we introduce the Differentiable Digital Signal Processing (DDSP) library, which enables direct integration of classic signal processing elements with deep learning methods. Focusing on audio synthesis, we achieve high-fidelity generation without the need for large autoregressive models or adversarial losses, demonstrating that DDSP enables utilizing strong inductive biases without losing the expressive power of neural networks. Further, we show that combining interpretable modules permits manipulation of each separate model component, with applications such as independent control of pitch and loudness, realistic extrapolation to pitches not seen during training, blind dereverberation of room acoustics, transfer of extracted room acoustics to new environments, and transformation of timbre between disparate sources. In short, DDSP enables an interpretable and modular approach to generative modeling, without sacrificing the benefits of deep learning. The library will be made available upon paper acceptance and we encourage further contributions from the community and domain experts.

Subfields: Computer Vision and Pattern Recognition, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Speech & Audio

In Signal Processing

Citations in the corpus

Symbolic Music Generation with Diffusion Models (2021, positive): “In the music domain, these techniques have led to improvements in expressive unconditional and conditional generation of both continuous audio (Engel et al., 2017b; 2019; Dhariwal et al., 2020; Engel et al., 2020), and discrete symbolic music (Huang et al.”
DiffWave: A Versatile Diffusion Model for Audio Synthesis (2020, background): “, 2020), and neural signal processing models (Wang et al., 2019; Engel et al., 2020; Ai & Ling, 2020).”
WaveGrad: Estimating Gradients for Waveform Generation (2020, neutral): “, 2020), as well as models inspired by digital signal processing (Ai & Ling, 2020; Engel et al., 2020), and the speech production mechanism (Juvela et al.”
SoundStream: An End-to-End Neural Audio Codec (2021, positive): “…loss for the generator is To promote ﬁdelity of the decoded signal ˆ x with respect to the original x we adopt two additional losses: i) a “feature” loss L feat G , computed in the feature space deﬁned by the discriminator(s) [15]; ii) a multi-scale spectral reconstruction loss L rec G [47].”
High-Fidelity Audio Compression with Improved RVQGAN (2023, methodology): “For the first, we use periods of [2, 3, 5, 7, 11], and for the second, we use window lengths [2048, 1024, 512], with a hop-length that is 1/4 the window length.”
A Survey on Neural Speech Synthesis (2021, background): “Other Vocoders Some works leverage neural-based source-filter model for waveform generation [375, 374, 371, 207, 146, 145, 75, 305, 407], aiming to achieve high voice quality while maintaining controllable speech generation.”
A Spectral Energy Distance for Parallel Speech Synthesis (2020, positive): “Our proposed spectral energy distance is closely related to other recent work in audio synthesis [35, 36, 9, 6], in that it is based on calculating distances between spectrograms, but we believe it is the ﬁrst to include a repulsive term between generated samples, and thus the ﬁrst proper scoring…”
It's Raw! Audio Generation with State-Space Models (2022, background): “, 2019) or differentiable digital signal processing (DDSP) models (Engel et al., 2020).”
RAVE: A variational autoencoder for fast and high-quality neural audio synthesis (2021, methodology): “The last sub-network is a noise synthesizer as proposed in Engel et al. (2019), and produces a multiband filtered noise added to the previous signal.”
FastSpeech 2: Fast and High-Quality End-to-End Text to Speech (2020, methodology): “Our Method To tackle the challenges above, we make several designs in the waveform decoder: 1) Considering the phase information is difficult to predict using a variance predictor [3], we introduce adversarial training in the waveform decoder to force it to implicitly recover the phase information b”
End-to-End Adversarial Text-to-Speech (2020, positive): “Spectrogram prediction losses have been used extensively for feed-forward audio prediction models [3, 16, 64, 67, 68, 69].”
MusicLM: Generating Music From Text (2023, background): “Symbolic representations of music (e.g., MIDI) can also be used to drive the generative process as a form of strong conditioning, as demonstrated by Huang et al. (2019); Hawthorne et al. (2019); Engel et al. (2020).”
Neural Analysis and Synthesis: Reconstructing Speech from Self-Supervised Representations (2021, background): “[17] proposed to integrate a harmonic plus noise model [38] and neural networks to produce natural audio signal.”
HooliGAN: Robust, High Quality Neural Vocoding (2020, methodology): “We use a multi-STFT (Short Time Fourier Transform) loss similar to DDSP [1] for both the output of the WaveNet ŷi and the WaveNet input Iij where we sum along the stacked axis and add the noise to get oi a 1d signal:”
Musika! Fast Infinite Waveform Music Generation (2022, unknown): “We leverage this by including the multi-scale spectral distance [7,28] in the objective of the decoder: where hop indicates a choice of hop _ size and fft _ size .”
Relative Positional Encoding for Transformers with Linear Complexity (2021, unknown): “Such considerations are close to the ideas proposed in (Engel et al., 2020).”
AudioLDM 2: Learning Holistic Audio Generation With Self-Supervised Pretraining (2023, background): “, 2023a), and specific types of sounds such as footsteps and violin sounds (Bresin et al., 2010; Engel et al., 2020).”
SingSong: Generating musical accompaniments from singing (2023, unknown): “(Hawthorne et al., 2018; Engel et al., 2019; Hawthorne et al., 2022b), or performance characteristics (Caillon & Esling, 2021; Engel et al., 2020).”
AudioLDM: Text-to-Audio Generation with Latent Diffusion Models (2023, background): “Additionally, as audio manipulations, such as style transfer (Engel et al., 2020; Pascual et al., 2022), are desired for some applications such as games, we explore and achieve various zero-shot text-guided audio manipulations with LDMs, which have not been demonstrated before.”
NANSY++: Unified Voice Synthesis with Neural Analysis and Synthesis (2022, methodology): “To train F 0 estimator in a self-supervised manner, we first adopt the auto-encoding approach of Engel et al. (2020b).”