It's Raw! Audio Generation with State-Space Models

By Karan Goel, Albert Gu, Chris Donahue, Christopher R'e. 2022

266 citations recorded by the source corpus.

Developing architectures suitable for modeling raw audio is a challenging problem due to the high sampling rates of audio waveforms. Standard sequence modeling approaches like RNNs and CNNs have previously been tailored to fit the demands of audio, but the resultant architectures make undesirable computational tradeoffs and struggle to model waveforms effectively. We propose SaShiMi, a new multi-scale architecture for waveform modeling built around the recently introduced S4 model for long sequence modeling. We identify that S4 can be unstable during autoregressive generation, and provide a simple improvement to its parameterization by drawing connections to Hurwitz matrices. SaShiMi yields state-of-the-art performance for unconditional waveform generation in the autoregressive setting. Additionally, SaShiMi improves non-autoregressive generation performance when used as the backbone architecture for a diffusion model. Compared to prior architectures in the autoregressive generation setting, SaShiMi generates piano and speech waveforms which humans find more musical and coherent respectively, e.g. 2x better mean opinion scores than WaveNet on an unconditional speech generation task. On a music generation task, SaShiMi outperforms WaveNet on density estimation and speed at both training and inference even when using 3x fewer parameters. Code can be found at https://github.com/HazyResearch/state-spaces and samples at https://hazyresearch.stanford.edu/sashimi-examples.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness (2022, background): “HiPPO [33] and its extensions, most notably S4 [29, 34, 35] projects the history on a polynomial basis, allowing accurate reconstruction of the history through state-space models.”
High Fidelity Neural Audio Compression (2022, background): “While many other approaches were explored (Yamamoto et al., 2020a; Kalchbrenner et al., 2018; Goel et al., 2022), the most relevant ones here are those based on Generative Adversarial Networks (GAN) (Kumar et al., 2019; Yamamoto et al., 2020a; Kong et al., 2020; Andreev et al., 2022) were able to…”
Mega-TTS: Zero-Shot Text-to-Speech at Scale with Intrinsic Inductive Bias (2023, background): “Previous works of autoregressive generative model mainly aim at waveform generation [41, 18] and continuous acoustic feature generation [59, 49].”
Efficiently Modeling Long Sequences with Structured State Spaces (2021, background): “Finally, we note that follow-up work found that this version of S4 can sometimes suﬀer from numerical instabilities when the A matrix has eigenvalues on the right half-plane [14].”
SingSong: Generating musical accompaniments from singing (2023, unknown): “Mehri et al., 2017; Donahue et al., 2019; Goel et al., 2022).”
MambaHSI: Spatial–Spectral Mamba for Hyperspectral Image Classification (2025, background): “Recently, state-space models (SSMs) [20], [21], in particular structured state-space sequence models (S4) [22], have emerged as an efficient and effective layer to construct deep networks, and have obtained superior performance in continuous long-sequence data analysis [22], [23].”
Simplified State Space Layers for Sequence Modeling (2022, background): “Extensions have shown good performance on raw audio generation [14] and classification of long movie clips [26].”
Resurrecting Recurrent Neural Networks for Long Sequences (2023, methodology): “Besides language, deep state-space models were also found successful for long video/audio understanding and generation tasks (Islam & Bertasius, 2022; Nguyen et al., 2022; Goel et al., 2022), and have attracted interest in biology (Bordin et al.”
U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentation (2024, background): “…state space sequence models (SSMs) [11,15], in particular structured state space sequence models (S4) [14], have emerged as an efficient and effective building block (or layer) for constructing deep networks, which obtained cutting-edge performance in continuous long-sequence data analysis [10,14].”
Wavefit: an Iterative and Non-Autoregressive Neural Vocoder Based on Fixed-Point Iteration (2022, methodology): “To reduce the number of iterations while maintaining the quality, existing studies of DDPMs have investigated the inference noise schedule [44], the use of adaptive prior [45, 46], the network architecture [47, 48], and/or the training strategy [49].”
SpecGrad: Diffusion Probabilistic Model based Neural Vocoder with Adaptive Noise Spectral Shaping (2022, background): “To reduce the number of iterations while maintaining the quality, the conventional studies have proposed a proper inference noise schedule [25, 26] and/or network architecture [27, 28].”
On the Parameterization and Initialization of Diagonal State Space Models (2022, methodology): “We use an autoregressive (AR) model to vary the setting; this causal setting more closely imitates autoregressive speech generation, where SSMs have shown recent promise [5].”
Diagonal State Spaces are as Effective as Structured State Spaces (2022, background): “Its remarkable abilities are not limited to text and images and carry over to tasks such as time-series forecasting, speech recognition and audio generation (Goel et al., 2022).”
On the link between conscious function and general intelligence in humans and machines (2022, background): “Research in generative models has progressed rapidly in the last decade, with unconditional models capable of creating ever more realistic images (Dhariwal & Nichol, 2021; Casanova et al., 2021) and audio (Kong et al., 2020; Goel et al., 2022).”
Diffusion-based Time Series Imputation and Forecasting with Structured State Space Models (2022, background): “Building on the S4 layer, the authors presented SaShiMi, a generative model architecture for sequence generation [16] obtained by combining S4 layers in a U-net-inspired configuration.”
Hungry Hungry Hippos: Towards Language Modeling with State Space Models (2022, background): “State space models have shown promise in modeling sequential data, including time series data [25], audio [22], and visual data [44].”
GAN You Hear Me? Reclaiming Unconditional Speech Synthesis from Diffusion Models (2022, methodology): “Finally, for SaShiMi+DiffWave, we modify the autoregressive SaShiMi code and combine it with DiffWave according to [7]; we train it on SC09 for 800k updates with the parameters in the original paper [7].”