SingSong: Generating musical accompaniments from singing

By Chris Donahue, Antoine Caillon, Adam Roberts, Ethan Manilow, Philippe Esling, A. Agostinelli, Mauro Verzetti, Ian Simon, Olivier Pietquin, Neil Zeghidour and 1 more. 2023

79 citations recorded by the source corpus.

We present SingSong , a system that generates instrumental music to accompany input vocals, potentially offering musicians and non-musicians alike an intuitive new way to create music featuring their own voice. To accomplish this, we build on recent developments in musical source separation and audio generation. Speciﬁcally, we apply a state-of-the-art source separation algorithm to a large corpus of music audio to produce aligned pairs of vocals and instrumental sources. Then, we adapt AudioLM (Borsos et al., 2022)—a state-of-the-art approach for unconditional audio generation—to be suitable for conditional “audio-to-audio” generation tasks, and train it on the source-separated (vocal, instrumental) pairs. In a pairwise comparison with the same vocal inputs, listeners expressed a signiﬁcant preference for instrumentals generated by SingSong compared to those from a strong retrieval baseline.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SoundStorm: Efficient Parallel Audio Generation (2023, neutral): “…this hierarchical approach has demonstrated remarkable results for speech (Kharitonov et al., 2023) and music modeling (Agostinelli et al., 2023; Donahue et al., 2023), the computational cost of modeling ﬂattened SoundStream tokens with self-attention scales quadratically with the sequence…”
Simple and Controllable Music Generation (2023, background): “In parallel, Donahue et al. [2023] follows a similar approach but for the task of singing to accompaniment generation.”
Textually Pretrained Speech Language Models (2023, background): “Donahue et al. [2023] proposed a similar modeling approach for the task of singing to accompaniment generation.”
TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition (2023, unknown): “Recently, this framework has also been applied to generating music from text descriptions [20], accompaniment for singing [21], and text to speech (TTS) [22].”