Context-Aware Prosody Correction for Text-Based Speech Editing

By Max Morrison, Lucas Rencker, Zeyu Jin, Nicholas J. Bryan, J. Cáceres, Bryan Pardo. 2021

36 citations recorded by the source corpus.

Text-based speech editors expedite the process of editing speech recordings by permitting editing via intuitive cut, copy, and paste operations on a speech transcript. A major drawback of current systems, however, is that edited recordings often sound unnatural because of prosody mismatches around edited regions. In our work, we propose a new context-aware method for more natural sounding text-based editing of speech. To do so, we 1) use a series of neural networks to generate salient prosody features that are dependent on the prosody of speech surrounding the edit and amenable to fine-grained user control 2) use the generated features to control a standard pitch-shift and time-stretch method and 3) apply a denoising neural network to remove artifacts induced by the signal manipulation to yield a high-fidelity result. We evaluate our approach using a subjective listening test, provide a detailed comparative analysis, and conclude several interesting insights.

Subfields: Artificial Intelligence, Experimental and Cognitive Psychology, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Tan Lee
2. Yu Ting Yeung
3. Daxin Tan
4. Hyeongju Kim
5. Xin Jiang
6. Jaesung Tae
7. Liqun Deng
8. Taesu Kim

In Signal Processing

1. Tan Lee
2. Yu Ting Yeung
3. Hyeongju Kim
4. Xin Jiang
5. Jaesung Tae
6. Liqun Deng
7. Taesu Kim
8. Xiao Chen

Citations in the corpus

• A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, neutral): “Morrison et al. (2021) propose to retrieve the mod-ified speech segments from other utterances of the same speaker and correct the prosody with a context-aware TD-PSOLA corrector (Moulines & Charpentier, 1990).”
• EditSpeech: A Text Based Speech Editing System Using Partial Inference and Bidirectional Fusion (2021, methodology): “In [5], context-aware prosody correction was applied in single-speaker English speech editing.”
• EdiTTS: Score-based Editing for Controllable Text-to-Speech (2021, methodology): “Previous works have used unit selection [12] or context-aware prosody correction [13], but their reliance on traditional algorithms such as concatenation or TD-PSLA [14] complicate the synthesis pipeline while also compromising output quality.”