Whispered and Lombard Neural Speech Synthesis

By Qiong Hu, T. Bleisch, P. Petkov, Tuomo Raitio, Erik Marchi, Varun Bharadhwaj Lakshminarasimhan. 2021

18 citations recorded by the source corpus.

It is desirable for a text-to-speech system to take into account the environment where synthetic speech is presented, and provide appropriate context-dependent output to the user. In this paper, we present and compare various approaches for generating different speaking styles, namely, normal, Lombard, and whisper speech, using only limited data. The following systems are proposed and assessed: 1) Pre-training and fine-tuning a model for each style. 2) Lombard and whisper speech conversion through a signal processing based approach. 3) Multi-style generation using a single model based on a speaker verification model. Our mean opinion score and AB preference listening tests show that 1) we can generate high quality speech through the pre-training/fine-tuning approach for all speaking styles. 2) Although our speaker verification (SV) model is not explicitly trained to discriminate different speaking styles, and no Lombard and whisper voice is used for pretrain this system, SV model can be used as style encoder for generating different style embeddings as input for Tacotron system. We also show that the resulting synthetic Lombard speech has a significant positive impact on intelligibility gain.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Xu Tan
  2. Tao Qin
  3. Tie-Yan Liu
  4. Frank K. Soong
  5. Sheng Zhao
  6. Yi Ren
  7. Zhou Zhao
  8. Lei He

In Signal Processing

  1. Xu Tan
  2. Tie-Yan Liu
  3. Tao Qin
  4. Frank K. Soong
  5. Sheng Zhao
  6. Yi Ren
  7. Zhou Zhao
  8. Lei He

Citations in the corpus