St-Bert: Cross-Modal Language Model Pre-Training for End-to-End Spoken Language Understanding

By Minjeong Kim, Gyuwan Kim, Sang-Woo Lee, Jung-Woo Ha. 2020

38 citations recorded by the source corpus.

Language model pre-training has shown promising results in various downstream tasks. In this context, we introduce a cross-modal pre-trained language model, called Speech-Text BERT (ST-BERT), to tackle end-to-end spoken language understanding (E2E SLU) tasks. Taking phoneme posterior and subword-level text as an input, ST-BERT learns a contextualized cross-modal alignment via our two proposed pre-training tasks: Cross-modal Masked Language Modeling (CM-MLM) and Cross-modal Conditioned Language Modeling (CM-CLM). Experimental results on three benchmarks present that our approach is effective for various SLU datasets and shows a surprisingly marginal performance degradation even when 1% of the training data are available. Also, our method shows further SLU performance gain via domain-adaptive pre-training with domain-specific speech-text pair data.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Yu Wu
4. Sanyuan Chen
5. Furu Wei
6. Long Zhou
7. Tom Ko
8. Zi-Hua Zhang

In Computer Vision and Pattern Recognition

1. Shujie Liu
2. Yu Wu
3. Sanyuan Chen
4. Zhihua Wei
5. Jinyu Li
6. Zi-Hua Zhang
7. Long Zhou
8. Tom Ko

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Although some speech-language pre-training work (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021) attempts to improve spoken language understanding tasks, these methods only focus on an encoder with task-specific layers for different tasks and do not pre-train a decoder for generation tasks…”
• SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, background): “Thanks to the rapid development of these single-modal pre-training works, researchers begin to pre-train a cross-modal model with both speech and text data (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021; Ao et al., 2022a; Bapna et al., 2021; Zhang et al., 2022b; Tang et al., 2022).”
• SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, background): “With the rapid development of unimodal pre-training in speechandnaturallanguageprocessing[1],[4],jointspeech-text pre-trainingobtainsmoreandmoreattentionfromresearch andindustrialcommunities[6],[15],[16],[17],[18],[19].”