SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training

By Zi-Hua Zhang, Long Zhou, Junyi Ao, Shujie Liu, Lirong Dai, Jinyu Li, Furu Wei. 2022

67 citations recorded by the source corpus.

The rapid development of single-modal pre-training has prompted researchers to pay more attention to cross-modal pre-training methods. In this paper, we propose a unified-modal speech-unit-text pre-training model, SpeechUT, to connect the representations of a speech encoder and a text decoder with a shared unit encoder. Leveraging hidden-unit as an interface to align speech and text, we can decompose the speech-to-text model into a speech-to-unit model and a unit-to-text model, which can be jointly pre-trained with unpaired speech and text data respectively. Our proposed SpeechUT is fine-tuned and evaluated on automatic speech recognition (ASR) and speech translation (ST) tasks. Experimental results show that SpeechUT gets substantial improvements over strong baselines, and achieves state-of-the-art performance on both the LibriSpeech ASR and MuST-C ST tasks. To better understand the proposed SpeechUT, detailed analyses are conducted. The code and pre-trained models are available at https://aka.ms/SpeechUT.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Zhuo Chen
4. Lei He
5. Yu Wu
6. Sanyuan Chen
7. Sheng Zhao
8. Yanqing Liu

Citations in the corpus

• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “We leverage the improved SpeechUT [Zhang et al., 2022c] as our speech recognition & translation model, which is a unified-modal speech-unit-text pre-training framework using hidden units as the modality bridge between speech and text.”
• DUB: Discrete Unit Back-translation for Speech Translation (2023, methodology): “…generative spoken language model (Lakhotia et al., 2021; Kharitonov et al., 2022), enhance speech representation (Chung et al., 2021; Meng et al., 2022; Chen et al., 2022; Wu et al., 2022; Zhang et al., 2022c), and model direct speech-to-speech translation (Lee et al., 2022; In-aguma et al., 2022).”
• Joint Pre-Training with Speech and Bilingual Text for Direct Speech to Speech Translation (2022, unknown): “Following the mixing mechanism proposed in [26], we also adapt it to improve alignment learning by randomly replacing part of H m with the corresponding unit embedding.”
• SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, methodology): “With the rapid development of unimodal pre-training in speechandnaturallanguageprocessing[1],[4],jointspeech-text pre-trainingobtainsmoreandmoreattentionfromresearch andindustrialcommunities[6],[15],[16],[17],[18],[19].”