Generative Spoken Dialogue Language Modeling

By Tu Nguyen, Evgeny Kharitonov, Jade Copet, Yossi Adi, Wei-Ning Hsu, Ali Mamdouh Elkahky, Paden Tomasello, Robin Algayres, Benoît Sagot, Abdel-rahman Mohamed and 1 more. 2022

155 citations recorded by the source corpus.

We introduce dGSLM, the first “textless” model able to generate audio samples of naturalistic spoken dialogues. It uses recent work on unsupervised spoken unit discovery coupled with a dual-tower transformer architecture with cross-attention trained on 2000 hours of two-channel raw conversational audio (Fisher dataset) without any text or labels. We show that our model is able to generate speech, laughter, and other paralinguistic signals in the two channels simultaneously and reproduces more naturalistic and fluid turn taking compared to a text-based cascaded model.1,2

Subfields: Artificial Intelligence, Dialogue & Conversational AI, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Wei-Ning Hsu
  2. Mohit Bansal
  3. James R. Glass
  4. Yoshua Bengio
  5. Dong Yu
  6. Gabriel Synnaeve
  7. Yossi Adi
  8. Jinyu Li

In Dialogue & Conversational AI

  1. Hung-yi Lee
  2. Shinji Watanabe
  3. Shang-Wen Li

In Speech & Audio

  1. Wei-Ning Hsu
  2. Yossi Adi
  3. Jinyu Li
  4. Abdel-rahman Mohamed
  5. Gabriel Synnaeve
  6. Shinji Watanabe
  7. Jade Copet
  8. Dong Yu

In Signal Processing

  1. Wei-Ning Hsu
  2. Jinyu Li
  3. Yossi Adi
  4. Shinji Watanabe
  5. Dong Yu
  6. Abdel-rahman Mohamed
  7. Emmanuel Dupoux
  8. Gabriel Synnaeve

Citations in the corpus