DUB: Discrete Unit Back-translation for Speech Translation
By Dong Zhang, Rong Ye, Tom Ko, Mingxuan Wang, Yaqian Zhou. 2023
34 citations recorded by the source corpus.
How can speech-to-text translation (ST) perform as well as machine translation (MT)? The key point is to bridge the modality gap between speech and text so that useful MT techniques can be applied to ST. Recently, the approach of representing speech with unsupervised discrete units yields a new way to ease the modality problem. This motivates us to propose Discrete Unit Back-translation (DUB) to answer two questions: (1) Is it better to represent speech with discrete units than with continuous features in direct ST? (2) How much benefit can useful MT techniques bring to ST? With DUB, the back-translation technique can successfully be applied on direct ST and obtains an average boost of 5.5 BLEU on MuST-C En-De/Fr/Es. In the low-resource language scenario, our method achieves comparable performance to existing methods that rely on large-scale external data. Code and models are available at https://github.com/0nutation/DUB.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
Citations in the corpus
- PolyVoice: Language Models for Speech to Speech Translation (2023, methodology): “For the speech synthesis part, we adopt the VALL-E X approach (Zhang et al., 2023) for the voice clone ability.”
- SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilities (2023, methodology): “…on generative spoken language models involves encoding the speech signal into a discrete representation (Baevski et al., 2020; Hsu et al., 2021; Zhang et al., 2023a) and modeling it with language models (Lakhotia et al., 2021; Borsos et al., 2022; Zhang et al., 2023d; Wang et al., 2023; Zhang…”