End-to-End Binaural Speech Synthesis

By Wen-Chin Huang, Dejan Markovic, Alexander Richard, Israel Dejene Gebru, Anjali Menon. 2022

15 citations recorded by the source corpus.

In this work, we present an end-to-end binaural speech synthesis system that combines a low-bitrate audio codec with a powerful binaural decoder that is capable of accurate speech binauralization while faithfully reconstructing environmental factors like ambient noise or reverb. The network is a modified vector-quantized variational autoencoder, trained with several carefully designed objectives, including an adversarial loss. We evaluate the proposed system on an internal binaural dataset with objective metrics and a perceptual study. Results show that the proposed approach matches the ground truth data more closely than previous methods. In particular, we demonstrate the ca-pability of the adversarial loss in capturing environment effects needed to create an authentic auditory scene.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dejan Markovic
2. Israel Dejene Gebru
3. Alexander Richard
4. Yi-Chiao Wu
5. Juergen Gall
6. Hilde Kuehne

In Signal Processing

1. Dejan Markovic
2. Israel Dejene Gebru
3. Alexander Richard
4. Yi-Chiao Wu

Citations in the corpus

• Audiodec: An Open-Source Streaming High-Fidelity Neural Audio Codec (2023, methodology): “Specifically, since both the encoder and decoder are easily replaceable, we can separately develop several specific encoders and decoders for any applications and easily integrate or switch among them for different real-world scenarios such as binaural rendering [28].”