Direct speech-to-speech translation with a sequence-to-sequence model

By Ye Jia, Ron J. Weiss, Fadi Biadsy, Wolfgang Macherey, Melvin Johnson, Z. Chen, Yonghui Wu. 2019

281 citations recorded by the source corpus.

We present an attention-based sequence-to-sequence neural network which can directly translate speech from one language into speech in another language, without relying on an intermediate text representation. The network is trained end-to-end, learning to map speech spectrograms into target spectrograms in another language, corresponding to the translated content (in a different canonical voice). We further demonstrate the ability to synthesize translated speech using the voice of the source speaker. We conduct experiments on two Spanish-to-English speech translation datasets, and find that the proposed model slightly underperforms a baseline cascade of a direct speech-to-text translation model and a text-to-speech synthesis model, demonstrating the feasibility of the approach on this very challenging task.

Subfields: Artificial Intelligence, Natural Language Processing, Machine Translation

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

  1. Tao Qin
  2. Xu Tan
  3. Tie-Yan Liu
  4. Yu Zhang
  5. Jinyu Li
  6. Wei-Ning Hsu
  7. Dong Yu
  8. Furu Wei

In Natural Language Processing

  1. Tao Qin
  2. Tie-Yan Liu
  3. Xu Tan
  4. Yu Zhang
  5. Jinyu Li
  6. Furu Wei
  7. Jiatao Gu
  8. Lei He

In Machine Translation

  1. Ye Jia
  2. Juan Pino
  3. Changhan Wang
  4. Michelle Tadmor Ramanovich
  5. Tal Remez
  6. Anne Wu
  7. Roi Pomerantz
  8. Yossi Adi

Citations in the corpus