Textually Pretrained Speech Language Models
By Michael Hassid, Tal Remez, Tu Nguyen, Itai Gat, Alexis Conneau, Felix Kreuk, Jade Copet, Alexandre Défossez, Gabriel Synnaeve, Emmanuel Dupoux and 2 more. 2023
112 citations recorded by the source corpus.
Speech language models (SpeechLMs) process and generate acoustic data only, without textual supervision. In this work, we propose TWIST, a method for training SpeechLMs using a warm-start from a pretrained textual language models. We show using both automatic and human evaluations that TWIST outperforms a cold-start SpeechLM across the board. We empirically analyze the effect of different model design choices such as the speech tokenizer, the pretrained textual model, and the dataset size. We find that model and dataset scale both play an important role in constructing better-performing SpeechLMs. Based on our observations, we present the largest (to the best of our knowledge) SpeechLM both in terms of number of parameters and training data. We additionally introduce two spoken versions of the StoryCloze textual benchmark to further improve model evaluation and advance future research in the field. We make speech samples, code and models publicly available: https://pages.cs.huji.ac.il/adiyoss-lab/twist/ .
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Yossi Adi
- Gabriel Synnaeve
- Shinji Watanabe
- Tal Remez
- Xuankai Chang
- Felix Kreuk
- Itai Gat
- Yaron Lipman
Citations in the corpus
- AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “SpeechLM [Hassid et al., 2023] also exploits the similarity between text and semantic tokens by initializing a decoder-only audio generator with the weights of a pretrained text-based language model.”
- Discrete Flow Matching (2024, background): “…as, code (Roziere et al., 2023; Li et al., 2023; Chen et al., 2021), biology (Zhang et al., 2024; Ferruz and Höcker, 2022; Madani et al., 2023), math (Romera-Paredes et al., 2024; Imani et al., 2023; Ahn et al., 2024), audio (Kreuk et al., 2022; Copet et al., 2024; Hassid et al., 2024) and more.”
- VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, methodology): “Moreover, we utilize textually pre-trained OPT [21] for better initialization inspired by [22] and leverage different speech tokens.”