AUDIT: Audio Editing by Following Instructions with Latent Diffusion Models

By Yuancheng Wang, Zeqian Ju, Xuejiao Tan, Lei He, Zhizheng Wu, Jiang Bian, Sheng Zhao. 2023

107 citations recorded by the source corpus.

Audio editing is applicable for various purposes, such as adding background sound effects, replacing a musical instrument, and repairing damaged audio. Recently, some diffusion-based methods achieved zero-shot audio editing by using a diffusion and denoising process conditioned on the text description of the output audio. However, these methods still have some problems: 1) they have not been trained on editing tasks and cannot ensure good editing effects; 2) they can erroneously modify audio segments that do not require editing; 3) they need a complete description of the output audio, which is not always available or necessary in practical scenarios. In this work, we propose AUDIT, an instruction-guided audio editing model based on latent diffusion models. Specifically, AUDIT has three main design features: 1) we construct triplet training data (instruction, input audio, output audio) for different audio editing tasks and train a diffusion model using instruction and input (to be edited) audio as conditions and generating output (edited) audio; 2) it can automatically learn to only modify segments that need to be edited by comparing the difference between the input and output audio; 3) it only needs edit instructions instead of full target audio descriptions as text input. AUDIT achieves state-of-the-art results in both objective and subjective metrics for several audio editing tasks (e.g., adding, dropping, replacement, inpainting, super-resolution). Demo samples are available at https://audit-demo.github.io/.

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Matt Le
2. Jiatong Shi
3. Jiang Bian
4. Bowen Shi
5. Dongchao Yang
6. Rongjie Huang
7. Xuejiao Tan
8. Andros Tjandra

In Signal Processing

1. Dongchao Yang
2. Rongjie Huang
3. Matt Le
4. Jiatong Shi
5. Jiang Bian
6. Bowen Shi
7. Xuejiao Tan
8. Andros Tjandra

Citations in the corpus

• Audiobox: Unified Audio Generation with Natural Language Prompts (2023, background): “Recent models also extends to more general editing, such as removal or addition of sound events with natural language instructions (Wang et al., 2023b; Liu et al., 2023d).”
• UniAudio: An Audio Foundation Model Toward Universal Audio Generation (2023, methodology): “AUDIT Wang et al. (2023d) is the SOTA model in audio edit task, which designs a data simulation strategy to get triplet training and test data ( e.g. , {audio, audio, text}).”