Audio-Visual Speech Codecs: Rethinking Audio-Visual Speech Enhancement by Re-Synthesis
By Karren D. Yang, Dejan Markovic, Steven Krenn, Vasu Agrawal, Alexander Richard. 2022
49 citations recorded by the source corpus.
Since facial actions such as lip movements contain significant information about speech content, it is not surprising that audio-visual speech enhancement methods are more accurate than their audio-only counterparts. Yet, state-of-the-art approaches still struggle to generate clean, realistic speech without noise artifacts and unnatural distortions in challenging acoustic environments. In this paper, we propose a novel audio-visual speech enhancement framework for high-fidelity telecommunications in AR/VR. Our approach leverages audio-visual speech cues to generate the codes of a neural speech codec, enabling efficient synthesis of clean, realistic speech from noisy signals. Given the importance of speaker-specific cues in speech, we focus on developing personalized models that work well for individual speakers. We demonstrate the efficacy of our approach on a new audio-visual speech dataset collected in an unconstrained, large vocabulary setting, as well as existing audio-visual datasets, outperforming speech enhancement baselines on both quantitative metrics and human evaluation studies. Please see the supplemental video for qualitative results11https://github.com/facebookresearch/facestar/releases/download/paper_materials/video.mp4.
Subfields: Computer Vision and Pattern Recognition, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Computer Vision and Pattern Recognition
- Dejan Markovic
- Chutong Meng
- Zhichao Huang
- Israel Dejene Gebru
- Tom Ko
- Alexander Richard
- Yi-Chiao Wu
- Juergen Gall
In Signal Processing
- Tom Ko
- Chutong Meng
- Zhichao Huang
- Dejan Markovic
- Israel Dejene Gebru
- Alexander Richard
- Yi-Chiao Wu
- Daniel Povey
Citations in the corpus
- RepCodec: A Speech Representation Codec for Speech Tokenization (2023, methodology): “, 2019), and Gumble-softmax (Yang et al., 2022) also work well for optimization of the VQ module.”
- Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, methodology): “For example, the neural codecs with softmax quantization [21], straight-through gradient, exponential moving average (EMA) [22, 23], and Gumble-softmax [31] have been recently proposed and work well for gradient propagation.”