Controllable Neural Prosody Synthesis
By Max Morrison, Zeyu Jin, Justin Salamon, Nicholas J. Bryan, Gautham J. Mysore. 2020
24 citations recorded by the source corpus.
Speech synthesis has recently seen significant improvements in fidelity, driven by the advent of neural vocoders and neural prosody generators. However, these systems lack intuitive user controls over prosody, making them unable to rectify prosody errors (e.g., misplaced emphases and contextually inappropriate emotions) or generate prosodies with diverse speaker excitement levels and emotions. We address these limitations with a user-controllable, context-aware neural prosody generator. Given a real or synthesized speech recording, our model allows a user to input prosody constraints for certain time frames and generates the remaining time frames from input text and contextual prosody. We also propose a pitch-shifting neural vocoder to modify input speech to match the synthesized prosody. Through objective and subjective evaluations we show that we can successfully incorporate user control into our prosody generation model without sacrificing the overall naturalness of the synthesized speech.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- A Survey on Neural Speech Synthesis (2021, background): “2) Extract the pitch and energy information from speech and extract duration from paired text and speech data [284, 286, 177, 155, 233, 359].”
- Review of end-to-end speech synthesis technology based on deep learning (2021, methodology): “A simple method to reduce exposure bias is scheduled sampling [13], in which acoustic feature frames of the current time step are predicted by using natural acoustic feature frames or those predicted by the previous time step with a certain probability [141, 155].”
- Chunked Autoregressive GAN for Conditional Waveform Synthesis (2021, background): “…audio waveform generation has seen remarkable improvements in fidelity and computational cost in recent years, benefiting applications such as text-to-speech (Kim et al., 2021), voice editing (Qian et al., 2020; Morrison et al., 2020), and controllable music generation (Dhari-wal et al., 2020).”
- Context-Aware Prosody Correction for Text-Based Speech Editing (2021, unknown): “C-DAR predicts a categorical distribution over 128 possible pitch values, which are evenly distributed between -4 and +4 standard deviations from the speaker’s average pitch in base-2 log-space.”