Speech-Language Pre-Training for End-to-End Spoken Language Understanding

By Yao Qian, Ximo Bian, Yu Shi, Naoyuki Kanda, Leo Shen, Zhen Xiao, Michael Zeng. 2021

54 citations recorded by the source corpus.

End-to-end (E2E) spoken language understanding (SLU) can infer semantics directly from speech signal without cascading an automatic speech recognizer (ASR) with a natural language understanding (NLU) module. However, paired utterance recordings and corresponding semantics may not always be available or sufficient to train an E2E SLU model in a real production environment. In this paper, we propose to unify a well-optimized E2E ASR encoder (speech) and a pre-trained language model encoder (language) into a transformer decoder. The unified speech-language pre-trained model (SLP) is continually enhanced on limited labeled data from a target domain by using a conditional masked language model (MLM) objective, and thus can effectively generate a sequence of intent, slot type, and slot value for given input speech in the inference. The experimental results on two public corpora show that our approach to E2E SLU is superior to the conventional cascaded method. It also outperforms the present state-of-the-art approaches to E2E SLU with much less paired data.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Yu Wu
4. Sanyuan Chen
5. Furu Wei
6. Long Zhou
7. Tom Ko
8. Zi-Hua Zhang

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Although some speech-language pre-training work (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021) attempts to improve spoken language understanding tasks, these methods only focus on an encoder with task-specific layers for different tasks and do not pre-train a decoder for generation tasks…”
• SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, background): “With the rapid development of unimodal pre-training in speechandnaturallanguageprocessing[1],[4],jointspeech-text pre-trainingobtainsmoreandmoreattentionfromresearch andindustrialcommunities[6],[15],[16],[17],[18],[19].”
• SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, background): “Thanks to the rapid development of these single-modal pre-training works, researchers begin to pre-train a cross-modal model with both speech and text data (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021; Ao et al., 2022a; Bapna et al., 2021; Zhang et al., 2022b; Tang et al., 2022).”