ELLA-V: Stable Neural Codec Language Modeling with Alignment-guided Sequence Reordering

By Ya-Zhen Song, Zhuo Chen, Xiaofei Wang, Ziyang Ma, Xie Chen. 2024

81 citations recorded by the source corpus.

The language model (LM) approach based on acoustic and linguistic prompts, such as VALL-E, has achieved remarkable progress in the field of zero-shot audio generation. However, existing methods still have some limitations: 1) repetitions, transpositions, and omissions in the output synthesized speech due to limited alignment constraints between audio and phoneme tokens; 2) challenges of fine-grained control over the synthesized speech with autoregressive (AR) language model; 3) infinite silence generation due to the nature of AR-based decoding, especially under the greedy strategy. To alleviate these issues, we propose ELLA-V, a simple but efficient LM-based zero-shot text-to-speech (TTS) framework, which enables fine-grained control over synthesized audio at the phoneme level. The key to ELLA-V is interleaving sequences of acoustic and phoneme tokens, where phoneme tokens appear ahead of the corresponding acoustic tokens. The experimental findings reveal that our model outperforms baselines in terms of accuracy and delivers more stable results using both greedy and sampling-based decoding strategies.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Xu Tan
3. Shujie Liu
4. Sheng Zhao
5. Zhuo Chen
6. Yanqing Liu
7. Tao Qin
8. Tie-Yan Liu

In Natural Language Processing

1. Xu Tan
2. Sheng Zhao
3. Jinyu Li
4. Yanqing Liu
5. Zhuo Chen
6. Hiroshi Saruwatari
7. Dongchao Yang
8. Naoyuki Kanda

In Speech & Audio

1. Xu Tan
2. Sheng Zhao
3. Jinyu Li
4. Yanqing Liu
5. Shujie Liu
6. Zhuo Chen
7. Dongchao Yang
8. Sanyuan Chen

In Signal Processing

1. Jinyu Li
2. Xu Tan
3. Shujie Liu
4. Sheng Zhao
5. Zhuo Chen
6. Sanyuan Chen
7. Yanqing Liu
8. Takuya Yoshioka

Citations in the corpus

• Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, neutral): “ELLA-V [47] and RALL-E [48] improved VALL-E’s robustness and stability by including speech-text alignment prediction in the decoding process.”
• VALL-E 2: Neural Codec Language Models are Human Parity Zero-Shot Text to Speech Synthesizers (2024, methodology): “ELLA-V [Song et al., 2024] and RALL-E [Xin et al., 2024] improved VALL-E’s robustness and stability by including speech-text alignment prediction into the decoding process.”
• SpeechX: Neural Codec Language Model as a Versatile Speech Transformer (2023, background): “…build on this work by expanding the tasks supported, enhancing robustness and quality (e.g., stabilizing the AR generation with duration control [41] and leveraging improved neural codecs [42]), and developing more advanced conditioning mechanisms (e.g., us-ing disentangled prosody and speaker…”
• Seed-TTS: A Family of High-Quality Versatile Speech Generation Models (2024, methodology): “…we conducted the same evaluation on several prior models, such as Jiang et al. [2023], Le et al. [2024], Wang et al. [2023b], Zhang et al. [2023], Song et al. [2024], Ren et al. [2020], Ju et al. [2024], and Shen et al. [2023], all of which produced CMOS results below -1, indicating a…”
• RALL-E: Robust Codec Language Modeling with Chain-of-Thought Prompting for Text-to-Speech Synthesis (2024, methodology): “Song et al. [26] proposed ELLA-V that interleaves the speech tokens with phonemes by inserting a phoneme token and a special EndOfPhone (EOP) token at the beginning and end of the speech tokens corresponding to the phoneme, respectively.”
• F5-TTS: A Fairytaler that Fakes Fluent and Faithful Speech with Flow Matching (2024, methodology): “ELLA-V [15] proposed a challenging set containing 100 difficult textual patterns evaluating the robustness of the TTS model.”