Synchronous Speech Recognition and Speech-to-Text Translation with Interactive Decoding

By Jiajun Zhang, Hao Xiong, Long Zhou, Zhongjun He, Hua Wu, Haifeng Wang, Chengqing Zong. 2019

73 citations recorded by the source corpus.

Speech-to-text translation (ST), which translates source language speech into target language text, has attracted intensive attention in recent years. Compared to the traditional pipeline system, the end-to-end ST model has potential benefits of lower latency, smaller model size, and less error propagation. However, it is notoriously difficult to implement such a model without transcriptions as intermediate. Existing works generally apply multi-task learning to improve translation quality by jointly training end-to-end ST along with automatic speech recognition (ASR). However, different tasks in this method cannot utilize information from each other, which limits the improvement. Other works propose a two-stage model where the second model can use the hidden state from the first one, but its cascade manner greatly affects the efficiency of training and inference process. In this paper, we propose a novel interactive attention mechanism which enables ASR and ST to perform synchronously and interactively in a single model. Specifically, the generation of transcriptions and translations not only relies on its previous outputs but also the outputs predicted in the other task. Experiments on TED speech translation corpora have shown that our proposed model can outperform strong baselines on the quality of speech translation and achieve better speech recognition performances as well.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Juan Pino
2. Changhan Wang
3. Yun Tang
4. Jiatao Gu
5. Anne Wu
6. Xutai Ma
7. Dmytro Okhonko
8. Wei-Ning Hsu

In Machine Translation

1. Juan Pino
2. Changhan Wang
3. Yun Tang
4. Anne Wu
5. Xutai Ma
6. Dmytro Okhonko
7. Jiatao Gu
8. Alexis Conneau

In Speech & Audio

1. Changhan Wang
2. Juan Pino
3. Yun Tang
4. Anne Wu
5. Xutai Ma
6. Jiatao Gu
7. Dmytro Okhonko
8. Alexis Conneau

Citations in the corpus

• Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, methodology): “It also integrates SIMULEVAL (Ma et al., 2020a) for simultaneous ST/MT metrics such as AL (average lagging) (Ma et al., 2019) and DAL (differentiable average Lagging) (Cherry and Foster, 2019).”
• Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, background): “And very recent works [12, 13, 14] have demonstrated the possibility of combining the two related tasks, ASR and ST, under the same encoderdecoder architecture to achieve better performance.”