SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding

By Yu-An Chung, Chenguang Zhu, Michael Zeng. 2021

72 citations recorded by the source corpus.

Spoken language understanding (SLU) requires a model to analyze input acoustic signal to understand its linguistic content and make predictions. To boost the models’ performance, various pre-training methods have been proposed to learn rich representations from large-scale unannotated speech and text. However, the inherent disparities between the two modalities necessitate a mutual analysis. In this paper, we propose a novel semi-supervised learning framework, SPLAT, to jointly pre-train the speech and language modules. Besides conducting a self-supervised masked language modeling task on the two individual modules using unpaired speech and text, SPLAT aligns representations from the two modules in a shared latent space using a small amount of paired speech and text. Thus, during fine-tuning, the speech module alone can produce representations carrying both acoustic information and contextual semantic knowledge of an input acoustic signal. Experimental results verify the effectiveness of our approach on various SLU tasks. For example, SPLAT improves the previous state-of-the-art performance on the Spoken SQuAD dataset by more than 10%.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shinji Watanabe
3. Dong Yu
4. Tara N. Sainath
5. Takaaki Hori
6. Rohit Prabhavalkar
7. Shujie Liu
8. Li Deng

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Although some speech-language pre-training work (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021) attempts to improve spoken language understanding tasks, these methods only focus on an encoder with task-specific layers for different tasks and do not pre-train a decoder for generation tasks…”
• A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “…(Morais et al. 2021; Yang et al. 2021), slot filling (Lai et al. 2021; Yang et al. 2021), sentiment analysis (Liu et al. 2020), question answering (Chung, Zhu, and Zeng 2020), named entity recognition (Shon et al. 2021; Borgholt et al. 2021a; Pasad et al. 2021) and speech translation (Bansal et…”
• End-to-End Speech Recognition: A Survey (2023, background): “We note that a number of recent works have explored approaches to combine speech and text modalities by attempting to implicitly or explicitly map them intoasharedspace[159],[335],[336],[337],[338],[339],[340], [341].”
• SpeechUT: Bridging Speech and Text with Hidden-Unit for Encoder-Decoder Based Speech-Text Pre-training (2022, background): “Thanks to the rapid development of these single-modal pre-training works, researchers begin to pre-train a cross-modal model with both speech and text data (Chung et al., 2021b; Kim et al., 2021; Qian et al., 2021; Ao et al., 2022a; Bapna et al., 2021; Zhang et al., 2022b; Tang et al., 2022).”