Direct simultaneous speech to speech translation
By Xutai Ma, Hongyu Gong, Danni Liu, Ann Lee, Yun Tang, Peng-Jen Chen, Wei-Ning Hsu, Kenneth Heafield, Philipp Koehn, Juan Pino. 2021
9 citations recorded by the source corpus.
We present a direct simultaneous speech-to-speech translation (Simul-S2ST) model, Furthermore, the generation of translation is independent from intermediate text representations. Our approach leverages recent progress on direct speech-to-speech translation with discrete units, in which a sequence of discrete representations, instead of continuous spectrogram features, learned in an unsupervised manner, are predicted from the model and passed directly to a vocoder for speech synthesis on-the-fly. We also introduce the variational monotonic multihead attention (V-MMA), to handle the challenge of inefficient policy learning in speech simultaneous translation. The simultaneous policy then operates on source speech features and target discrete units. We carry out empirical studies to compare cascaded and direct approach on the Fisher Spanish-English and MuST-C English-Spanish datasets. Direct simultaneous model is shown to outperform the cascaded model by achieving a better tradeoff between translation quality and latency.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
- Michelle Tadmor Ramanovich
- Ye Jia
- Tal Remez
- Roi Pomerantz
- Yeting Jia
- Xinjian Li
- Yu Zhang
- Qianqian Dong
Citations in the corpus
- AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “There are other cascaded S2ST systems that utilize learned discrete speech representations as an intermediate representation [Tjandra et al., 2019, Zhang et al., 2021, Lee et al., 2022, Ma et al., 2021, Lee et al., 2021].”
- Translatotron 2: High-quality direct speech-to-speech translation with voice preservation (2021, methodology): “…works on S2ST without relying on intermediate text representation are emerging, such as end-to-end direct S2ST (Jia et al., 2019b; Kano et al., 2021) and cascade S2ST based on discrete speech representation (Tjandra et al., 2019; Zhang et al., 2021; Lee et al., 2022; 2021a; Ma et al., 2021).”