Learning Shared Semantic Space for Speech-to-Text Translation

By Chi Han, Mingxuan Wang, Heng Ji, Lei Li. 2021

90 citations recorded by the source corpus.

Having numerous potential applications and great impact, end-to-end speech translation (ST) has long been treated as an independent task, failing to fully draw strength from the rapid advances of its sibling - text machine translation (MT). With text and audio inputs represented differently, the modality gap has rendered MT data and its end-to-end models incompatible with their ST counterparts. In observation of this obstacle, we propose to bridge this representation gap with Chimera. By projecting audio and text features to a common semantic representation, Chimera unifies MT and ST tasks and boosts the performance on ST benchmarks, MuST-C and Augmented Librispeech, to a new state-of-the-art. Specifically, Chimera obtains 27.1 BLEU on MuST-C EN-DE, improving the SOTA by a +1.9 BLEU margin. Further experimental analyses demonstrate that the shared semantic space indeed conveys common knowledge between these two tasks and thus paves a new way for augmenting training resources across modalities. Code, data, and resources are available at https://github.com/Glaciohound/Chimera-ST.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Jinyu Li
2. Shujie Liu
3. Xipeng Qiu
4. Furu Wei
5. Yu Wu
6. Long Zhou
7. Tom Ko
8. Tianyang Lin

Citations in the corpus

• SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “Besides, a series of research work begins to investigate joint text and speech training (Han et al., 2021; Ye et al., 2021; Tang et al., 2021a; Zheng et al., 2021; Tang et al., 2021b), but they are mainly designed for speech to text tasks.”
• DUB: Discrete Unit Back-translation for Speech Translation (2023, background): “…not rely on intermediate transcription has received considerable attention due to its potential applications in unwritten language scenarios and various domains (Bérard et al., 2016; Sung et al., 2019; Han et al., 2021; Papi et al., 2021; Fang et al., 2022; Ye et al., 2022; Cheng et al., 2023).”
• A Survey of Transformers (2021, background): “, visual question answering [55, 75, 77, 125], visual commonsense reasoning [75, 125], caption generation [22, 81, 128], speech-to-text translation [46] and text-to-image generation [29, 81, 107].”