High Fidelity Speech Regeneration with Application to Speech Enhancement

By Adam Polyak, Lior Wolf, Yossi Adi, Ori Kabeli, Yaniv Taigman. 2021

21 citations recorded by the source corpus.

Speech enhancement has seen great improvement in recent years mainly through contributions in denoising, speaker separation, and dereverberation methods that mostly deal with environmental effects on vocal audio. To enhance speech beyond the limitations of the original signal, we take a regeneration approach, in which we recreate the speech from its essence, including the semi-recognized speech, prosody features, and identity. We propose a wav-to-wav generative model for speech that can generate 24khz speech in a real-time manner and which utilizes a compact speech representation, composed of ASR and identity features, to achieve a higher level of intelligibility. Inspired by voice conversion methods, we train to augment the speech characteristics while preserving the identity of the source using an auxiliary identity network. Perceptual acoustic metrics and subjective tests show that the method obtains valuable improvements over recent baselines.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Speech Resynthesis from Discrete Disentangled Self-Supervised Representations (2021, methodology): “These representations are being used together with the identity and prosodic information for voice conversion [24, 25, 26].”
On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “More work is needed to adapt these architectures to the richness and variability of the speech signal (see Polyak et al., 2021a, for ﬁrst steps towards integrating prosody into discrete units).”
Universal Speech Enhancement with Score-based Diffusion (2022, background): “With this in mind, a number of works have recently started to zoom out the focus in order to embrace more realistic situations and tackle noise and reverberation at the same time (Su et al., 2021; 2019; Polyak et al., 2021).”
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement (2022, methodology): “Second, regardless of the training objective (regression [40], adversarial [43, 45, 50], or score-matching [48]), previous universal models predict reference clean speech directly.”
Direct simultaneous speech to speech translation (2021, methodology): “Lee et al. (2021) introduce a modiﬁed version of the HiFi-GAN neural vocoder (Polyak et al., 2021) for unit-to-waveform conversion.”