SEANet: A Multi-modal Speech Enhancement Network

By Marco Tagliasacchi, Yunpeng Li, Karolis Misiunas, Dominik Roblek. 2020

117 citations recorded by the source corpus.

We explore the possibility of leveraging accelerometer data to perform speech enhancement in very noisy conditions. Although it is possible to only partially reconstruct user's speech from the accelerometer, the latter provides a strong conditioning signal that is not influenced from noise sources in the environment. Based on this observation, we feed a multi-modal input to SEANet (Sound EnhAncement Network), a wave-to-wave fully convolutional model, which adopts a combination of feature losses and adversarial losses to reconstruct an enhanced version of user's speech. We trained our model with data collected by sensors mounted on an earbud and synthetically corrupted by adding different kinds of noise sources to the audio signal. Our experimental results demonstrate that it is possible to achieve very high quality results, even in the case of interfering speech at the same level of loudness. A sample of the output produced by our model is available at this https URL.

Subfields: Artificial Intelligence, Multimodal Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

SoundStream: An End-to-End Neural Audio Codec (2021, methodology): “For the wave-based discriminator, we use the same multi-resolution convolutional discriminator proposed in [15] and adopted in [45].”
High Fidelity Neural Audio Compression (2022, background): “…in various audio-related tasks, e.g., source separation and enhancement (Défossez et al., 2019; Defossez et al., 2020), neural vocoders (Kumar et al., 2019; Kong et al., 2020), audio codec (Zeghidour et al., 2021), and artiﬁcial bandwidth extension (Tagliasacchi et al., 2020; Li et al., 2021).”
FunCodec: A Fundamental, Reproducible and Integrable Open-Source Toolkit for Neural Speech Codec (2023, methodology): “While SoundStream utilizes streaming SEANet [7,8] as its encoder and decoder, Encodec incorporates extra LSTM [9] layers and a Demos are available at: https://funcodec.github.io Transformer-based language model [10] to improve the sequence modeling ability.”
AudioLM: A Language Modeling Approach to Audio Generation (2022, neutral): “Recent audio synthesis models have achieved nearly veridical signal quality by leveraging methods such as autoregressive waveform modeling [1], [2], adversarial training [3]–[5] or diffusion [6], [7].”
Real-Time Speech Frequency Bandwidth Extension (2020, methodology): “Our model is trained using a combination of adversarial and reconstruction objectives, deﬁned by a discriminator learned concurrently with the generator, with a design similar to the audio-only version of SEANet (Sound Enhancement Network) that we introduced in our previous work [1].”
Wavefit: an Iterative and Non-Autoregressive Neural Vocoder Based on Fixed-Point Iteration (2022, methodology): “[64] M. Tagliasacchi, Y. Li, K. Misiunas, and D. Roblek, “SEANet: A multi-modal speech enhancement network,” in Proc.”
HiFi++: a Unified Framework for Neural Vocoding, Bandwidth Extension and Speech Enhancement (2022, methodology): “It is noteworthy that we signiﬁcantly outperform the SEANet (Tagliasacchi et al., 2020) model, which is trained in a similar adversarial manner and has a larger number of parameters, but does not take into account spectral information.”
SpeechPainter: Text-conditioned Speech Inpainting (2022, background): “Also relevant, but not directly related to our work, are methods that enhance speech based on modalities other than text: using facial crops of the target speaker, denoising [6], inpainting speech [15] or separating a target speaker from a mixture [4, 20], and enhancing speech based on accelerometer”
LauraGPT: Listen, Attend, Understand, and Regenerate Audio with GPT (2023, methodology): “Our codec model shares a similar architecture as Encodec (Défossez et al., 2022), which consists of a convolutional recurrent encoder/decoder network (Tagliasacchi et al., 2020) and a residual vector quantizer (RVQ) (Vasuki & Vanathi, 2006).”
FoundationTTS: Text-to-Speech for ASR Customization with Generative Language Model (2023, methodology): “Neural codec models have achieved close to recording quality by leveraging methods such as autoregressive waveform modeling [34, 35, 36] , adversarial training [37, 14, 38, 39].”