Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining

By Cheng-I Lai, Yung-Sung Chuang, Hung-yi Lee, Shang-Wen Li, James R. Glass. 2020

65 citations recorded by the source corpus.

Much recent work on Spoken Language Understanding (SLU) is limited in at least one of three ways: models were trained on oracle text input and neglected ASR errors, models were trained to predict only intents without the slot values, or models were trained on a large amount of in-house data. In this paper, we propose a clean and general framework to learn semantics directly from speech with semi-supervision from transcribed or untranscribed speech to address these issues. Our framework is built upon pretrained end-to-end (E2E) ASR and self-supervised language models, such as BERT, and fine-tuned on a limited amount of target SLU data. We study two semi-supervised settings for the ASR component: supervised pretraining on transcribed speech, and unsupervised pretraining by replacing the ASR encoder with self-supervised speech representations, such as wav2vec. In parallel, we identify two essential criteria for evaluating SLU models: environmental noise-robustness and E2E semantics evaluation. Experiments on ATIS show that our SLU framework with speech as input can perform on par with those using oracle text as input in semantics understanding, even though environmental noise is present and a limited amount of labeled semantics data is available for training.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “…have investigated these SSL models’ capabilities on tasks including phoneme classiﬁcation [11, 7], speaker identiﬁcation [7, 8], speaker ver-iﬁcation [17], emotion recognition [15], ASR [9, 12, 10, 16], speech translation [7], spoken language understanding [18], voice conversion [19] and TTS [20].”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “…et al. 2019), speaker identiﬁcation (Fan et al. 2021), emotion recognition (Pascual et al. 2019), speech translation (Chung et al. 2019), spoken language understanding (Lai et al. 2020), voice conversion (Lin et al. 2021) and text-to-speech translation ( ´Alvarez, Pascual, and Bonafonte 2019).”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, methodology): “2021), slot filling (Lai et al. 2021; Yang et al. 2021), sentiment analysis (Liu et al.”
Textless Speech Emotion Conversion using Decomposed and Discrete Representations (2021, background): “…learning aims to distill features useful for downstream tasks, such as phone discrimination (Kharitonov et al., 2021b; Schneider et al., 2019) and semantic prediction (Lai et al., 2021; Wu et al., 2020), by constructing pretext tasks that can exploit large quantities of unlabeled speech.”
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022, unknown): “, 2021a), spoken language understanding (Lai et al., 2021), and text-tospeech (Álvarez et al.”