A General Multi-Task Learning Framework to Leverage Text Data for Speech to Text Tasks

By Yun Tang, Juan Pino, Changhan Wang, Xutai Ma, Dmitriy Genzel. 2020

82 citations recorded by the source corpus.

Attention-based sequence-to-sequence modeling provides a powerful and elegant solution for applications that need to map one sequence to a different sequence. Its success heavily relies on the availability of large amounts of training data. This presents a challenge for speech applications where labelled speech data is very expensive to obtain, such as automatic speech recognition (ASR) and speech translation (ST). In this study, we propose a general multi-task learning framework to leverage text data for ASR and ST tasks. Two auxiliary tasks, a denoising autoencoder task and machine translation task, are proposed to be co-trained with ASR and ST tasks respectively. We demonstrate that representing text input as phoneme sequences can reduce the difference between speech and text inputs, and enhance the knowledge transfer from text corpora to the speech to text tasks. Our experiments show that the proposed method achieves a relative 10~15% word error rate reduction on the English LIBRISPEECH task compared with our baseline, and improves the speech translation quality on the MUST-C tasks by 3.6~9.2 BLEU.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Furu Wei
3. Juan Pino
4. Changhan Wang
5. Shujie Liu
6. Michael Auli
7. Alexei Baevski
8. Alexis Conneau

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Besides, a series of research work begins to investigate joint text and speech training (Han et al., 2021; Ye et al., 2021; Tang et al., 2021a; Zheng et al., 2021; Tang et al., 2021b), but they are mainly designed for speech to text tasks.”
• Multilingual Speech Translation from Efficient Finetuning of Pretrained Models (2021, methodology): “Parameters in those 12 layers are shared between the two encoders during joint training (Tang et al., 2021b).”
• Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task (2021, background): “The input text tokens are converted into their corresponding pronunciation form as phoneme sequences (Tang et al., 2021; Renduchintala et al., 2018).”
• End-to-end Speech Translation via Cross-modal Progressive Training (2021, methodology): “Existing approaches to this problem can be grouped into two categories: a) multi-task supervision with the speech-transcript-translation triple data [1, 2]; b) pretraining with external large-scale MT parallel text data [3, 4, 5, 6].”
• Data Augmentation for ASR Using TTS Via a Discrete Representation (2021, background): “Another direction that has been pursued is incorporation of unpaired text data into (part of) the ASR model training [8, 9, 10, 11, 12].”
• Learning Shared Semantic Space for Speech-to-Text Translation (2021, unknown): “, 2020), multi-task training (Vydana et al., 2021; Le et al., 2020; Tang et al., 2021), meta-learning (Indurthi et al.”