Generative Spoken Dialogue Language Modeling
By Tu Nguyen, Evgeny Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Mamdouh Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdel-rahman Mohamed and 1 more. 2022
155 citations recorded by the source corpus.
We introduce dGSLM, the first “textless” model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter, and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn taking compared to a text-based cascaded model.1,2
Subfields: Artificial Intelligence, Dialogue & Conversational AI, Speech & Audio, Signal Processing
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
In Dialogue & Conversational AI
In Speech & Audio
- Wei-Ning Hsu
- Yossi Adi
- Jinyu Li
- Abdel-rahman Mohamed
- Gabriel Synnaeve
- Shinji Watanabe
- Jade Copet
- Dong Yu
In Signal Processing
- Wei-Ning Hsu
- Jinyu Li
- Yossi Adi
- Shinji Watanabe
- Dong Yu
- Abdel-rahman Mohamed
- Emmanuel Dupoux
- Gabriel Synnaeve
Citations in the corpus
- High Fidelity Neural Audio Compression (2022, methodology): “Similar methods were applied to speech resynthesis (Polyak et al., 2021), speech emotion conversion (Kreuk et al., 2021), spoken dialog system (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2021a;b; Popuri et al., 2022).”
- Voicebox: Text-Guided Multilingual Universal Speech Generation at Scale (2023, methodology): “The GSLM-family [36, 28, 41] are textless language models built upon HuBERT units [22] for speech continuation without using text.”
- AudioGen: Textually Guided Audio Generation (2022, background): “…modeling (Lakho-2 tia et al., 2021; Kharitonov et al., 2022a; Borsos et al., 2022), multi-stream processing (Kharitonov et al., 2022b), speech emotion conversion (Kreuk et al., 2022), spoken dialuge (Nguyen et al., 2022), and speech-to-speech translation (Lee et al., 2022a;b; Popuri et al., 2022).”
- Make-A-Voice: Unified Voice Synthesis With Discrete Representation (2023, background): “SpeechDLM [28] leverages the HuBERT representation and introduces the end-to-end spoken language model for dialogue.”
- Generative Pre-training for Speech with Flow Matching (2023, background): “The key difference is the goal of SLMs is to discover the underlying text for textless language processing (Nguyen et al., 2022).”
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023, methodology): “Generative Spoken Language Model Discrete self-supervised representation based spoken generative language modeling is making remarkable progress on large-scale speech dataset training (Nguyen et al., 2022).”
- AudioGPT: Understanding and Generating Speech, Music, Sound, and Talking Head (2023, background): “Recently, Nguyen et al. (2023) leverage the success of discrete representation and introduce the first end-to-end generative spoken dialogue language model.”
- Textually Pretrained Speech Language Models (2023, methodology): “Such a modeling framework additionally allows for capturing and modeling prosodic features [Kharitonov et al., 2021], as well as speaker identity [Borsos et al., 2022], or even natural dialogues [Nguyen et al., 2022].”
- Self-Supervised Speech Representation Learning: A Review (2022, background): “] → POWER) and continues using words in the same general mood (dark → BLACKNESS)6 and has been extended to model and generate dialogues [332].”
- textless-lib: a Library for Textless Spoken Language Processing (2022, background): “…textless-lib we envision improving performance of the existing building blocks, adding new example tasks (e.g., translation (Lee et al., 2021b) or dialog (Nguyen et al., 2022)), extending the set of provided pre-trained models, and introducing the possibility of training the different components.”