Textless Speech-to-Speech Translation on Real Data

By Ann Lee, Hongyu Gong, Paul-Ambroise Duquenne, Holger Schwenk, Peng-Jen Chen, Changhan Wang, Sravya Popuri, Juan Pino, Jiatao Gu, Wei-Ning Hsu. 2021

187 citations recorded by the source corpus.

We present a textless speech-to-speech translation (S2ST) system that can translate speech from one language into another language and can be built without the need of any text data. Different from existing work in the literature, we tackle the challenge in modeling multi-speaker target speech and train the systems with real-world S2ST data. The key to our approach is a self-supervised unit-based speech normalization technique, which finetunes a pre-trained speech encoder with paired audios from multiple speakers and a single reference speaker to reduce the variations due to accents, while preserving the lexical content. With only 10 minutes of paired data for speech normalization, we obtain on average 3.2 BLEU gain when training the S2ST model on the VoxPopuli S2ST dataset, compared to a baseline trained on un-normalized speech target. We also incorporate automatically mined S2ST data and show an additional 2.0 BLEU gain. To our knowledge, we are the first to establish a textless S2ST technique that can be trained with real-world data and works for multiple language pairs.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Jinyu Li
  2. Dong Yu
  3. Wei-Ning Hsu
  4. Furu Wei
  5. Shinji Watanabe
  6. James R. Glass
  7. Yoshua Bengio
  8. Tao Qin

In Machine Translation

  1. Juan Pino
  2. Changhan Wang
  3. Ye Jia
  4. Yossi Adi
  5. Ann Lee
  6. Michelle Tadmor Ramanovich
  7. Wei-Ning Hsu
  8. Tal Remez

In Speech & Audio

  1. Jinyu Li
  2. Wei-Ning Hsu
  3. Yossi Adi
  4. Shinji Watanabe
  5. Dong Yu
  6. Abdel-rahman Mohamed
  7. Lei He
  8. Rongjie Huang

Citations in the corpus