SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data

By Zi-Hua Zhang, Sanyuan Chen, Long Zhou, Yu Wu, Shuo Ren, Shujie Liu, Zhuoyuan Yao, Xun Gong, Lirong Dai, Jinyu Li and 1 more. 2022

78 citations recorded by the source corpus.

How to boost speech pre-training with textual data is an unsolved problem due to the fact that speech and text are very different modalities with distinct characteristics. In this paper, we propose a cross-modal Speech and Language Model (SpeechLM) to explicitly align speech and text pre-training with a pre-defined unified discrete representation. Specifically, we introduce two alternative discrete tokenizers to bridge the speech and text modalities, including phoneme-unit and hidden-unit tokenizers, which can be trained using unpaired speech or a small amount of paired speech-text data. Based on the trained tokenizers, we convert the unlabeled speech and text data into tokens of phoneme units or hidden units. The pre-training objective is designed to unify the speech and the text into the same discrete semantic space with a unified Transformer network. We evaluate SpeechLM on various spoken language processing tasks including speech recognition, speech translation, and universal representation evaluation framework SUPERB, demonstrating significant improvements on content-related tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Zhuo Chen
4. Lei He
5. Yu Wu
6. Sanyuan Chen
7. Sheng Zhao
8. Yanqing Liu

Citations in the corpus

• Speak Foreign Languages with Your Own Voice: Cross-Lingual Neural Codec Language Modeling (2023, positive): “Inspired by SpeechLM [Zhang et al., 2022b] which explores different choices of units, we improve SpeechUT by replacing the clustering-based hidden units with phonemes.”
• SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, methodology): “One category of these works focuses on pre-training a unified encoder model for spoken language understanding (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021; Zhang et al., 2022a).”