SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering

By Yung-Sung Chuang, Chi-Liang Liu, Hung-yi Lee, Lin-Shan Lee. 2019

108 citations recorded by the source corpus.

While various end-to-end models for spoken language understanding tasks have been explored recently, this paper is probably the first known attempt to challenge the very difficult task of end-to-end spoken question answering (SQA). Learning from the very successful BERT model for various text processing tasks, here we proposed an audio-and-text jointly learned SpeechBERT model. This model outperformed the conventional approach of cascading ASR with the following text question answering (TQA) model on datasets including ASR errors in answer spans, because the end-to-end model was shown to be able to extract information out of audio data before ASR produced errors. When ensembling the proposed end-to-end model with the cascade architecture, even better performance was achieved. In addition to the potential of end-to-end SQA, the SpeechBERT can also be considered for many other spoken language understanding tasks just as BERT for many text processing tasks.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Zhiyuan Liu
2. Minlie Huang
3. Xipeng Qiu
4. Jinyu Li
5. Furu Wei
6. Yang Liu
7. Maosong Sun
8. Jie Tang

Citations in the corpus

• Pre-Trained Models: Past, Present and Future (2021, methodology): “SpeechBERT (Chuang et al., 2019) first encodes the continuous audio signal into several phonetic-semantic word embed-dings, and then uses MLM on both text and audio modalities as pre-training tasks.”
• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “…representation learning has also been investigated and shown promising results, benefiting from richly learned representations (Chung and Glass, 2018; Chuang et al., 2020; Song et al., 2019; Baevski et al., 2020; Wang et al., 2021; Hsu et al., 2021; Chung et al., 2021a), such as wav2vec 2.0…”
• Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation (2021, neutral): “…existing methods can only handle one modality, either text or speech, while joint acoustic and text representation is desired for many end-to-end spoken language processing tasks, such as spoken question answering (Chuang et al., 2019) and end-to-end speech-to-text translation (Liu et al., 2020b).”
• Speech-Language Pre-Training for End-to-End Spoken Language Understanding (2021, background): “, VLP [9] and SpeechBERT [10] proposed to build two-modality joint embeddings for image captioning, spoken question answering, etc.”
• MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation (2020, unknown): “Although current speech-related pre-training frameworks (Chuang et al., 2019; Wang et al., 2020b) indeed relieve certain needs of large scale parallel training corpus for E2E-ST, all of these pre-training methods still need intense transcription annotation for the source speech.”