Speech Model Pre-training for End-to-End Spoken Language Understanding

By Loren Lugosch, Mirco Ravanelli, Patrick Ignoto, Vikrant Singh Tomar, Yoshua Bengio. Interspeech, 2019

402 citations recorded by the source corpus.

Whereas conventional spoken language understanding (SLU) systems map speech to text, and then text to intent, end-to-end SLU systems map speech directly to intent through a single trainable model. Achieving high accuracy with these end-to-end models without a large amount of training data is difficult. We propose a method to reduce the data requirements of end-to-end SLU in which the model is first pre-trained to predict words and phonemes, thus learning good features for SLU. We introduce a new SLU dataset, Fluent Speech Commands, and show that our method improves performance both when the full dataset is used for training and when only a small subset is used. We also describe preliminary experiments to gauge the model's ability to generalize to new phrases not heard during training.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/24d7b1487202e3aaf329df3d8135ae6eabefaa45

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. James R. Glass
2. Hung-yi Lee
3. Yoshua Bengio
4. Dong Yu
5. Jinyu Li
6. Shinji Watanabe
7. Yu-An Chung
8. Li Deng

Citations in the corpus

• Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, methodology): “Representation extraction: We use LibriSpeech alignments generated using the Montreal forced aligner [41, 42] to define phone and word segments.”
• MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, methodology): “…Understanding Resource Package (SLURP) (Bastianelli et al., 2020), allowing for full end-to-end Spoken Language Understanding (SLU) evaluation similar to the Fluent Speech Commands dataset (Lugosch et al., 2019) and Chinese Audio-Textual Spoken Language Understanding (CATSLU) (Zhu et al., 2019).”
• St-Bert: Cross-Modal Language Model Pre-Training for End-to-End Spoken Language Understanding (2020, methodology): “End-to-end spoken language understanding (E2E SLU) has recently shown promising results [1, 2, 3].”
• FrugalML: How to Use ML Prediction APIs More Accurately and Cheaply (2020, positive): “The commands in FLUENT are typically a phrase (e.g., “turn on the light” or “turn down the music”).”
• Self-Supervised Speech Representation Learning: A Review (2022, methodology): “VoxCeleb1 [236] for SID/ASV (automatic speaker verification) tasks, FSC (Fluent Speech Commands) [237] for IC (intent classification), QUESST (QUESST 2014) [238] for QbE (query by example), LS En-Fr [239] and CoVoST-2 [240] for ST (speech translation), and ALFFA and OpenSLR-multi for multilingual AS”
• SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering (2019, background): “2 to speed up the process, while have the text word embedding unfrozen to keep the joint audio-and-text embeddings flexible, considering the earlier experiences reported for end-to-end SLU [31].”
• Speech-Language Pre-Training for End-to-End Spoken Language Understanding (2021, methodology): “In the pre-training stage, we continually train the SLP model with the speech embeddings from the encoder of the AED-based ASR model and the tokenized transcription by using FSC or ATIS dataset.”
• SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “We use the Fluent Speech Commands [28] dataset, where each utterance is tagged with three intent labels: action, object, and location.”
• Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systems (2021, background): “However, such endto-end models are data-hungry and suffer from lack of training data (Lugosch et al. 2019; Li et al. 2020a).”
• SLURP: A Spoken Language Understanding Resource Package (2020, methodology): “The standard metric for evaluating E2E-SLU is accuracy, which is defined as “the accuracy of all slots for an utterance taken together – that is, if the predicted intent differs from the true intent in even one slot, the prediction is deemed incorrect” (Lugosch et al., 2019b).”
• Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining (2020, methodology): “…speech to alleviate: (1) the need for a large amount of in-house, homogenous data [2, 7, 8, 17], (2) the limitation of only intent classification [8, 9, 13] by predicting text, slots and intents, and (3) any additional manipulation on labels or loss, such as label projection [26], output…”
• SpeechBrain: A General-Purpose Speech Toolkit (2021, background): “There are currently recipes for three open-source SLU datasets with different levels of complexity: Fluent Speech Commands (FSC) [54], Timers and Such [50], and SLURP [53].”