An Attentional Model for Speech Translation Without Transcription

By Long Duong, Antonios Anastasopoulos, David Chiang, Steven Bird, Trevor Cohn. 2016

156 citations recorded by the source corpus.

For many low-resource languages, spoken language resources are more likely to be annotated with translations than transcriptions. This bilingual speech data can be used for word-spotting, spoken document retrieval, and even for documentation of endangered languages. We experiment with the neural, attentional model applied to this data. On phoneto-word alignment and translation reranking tasks, we achieve large improvements relative to several baselines. On the more challenging speech-to-word alignment task, our model nearly matches GIZA++’s performance on gold transcriptions, but without recourse to transcriptions or to a lexicon.

Subfields: Artificial Intelligence, Machine Translation, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Machine Translation

In Speech & Audio

Citations in the corpus

Fairseq S2T: Fast Speech-to-Text Modeling with Fairseq (2020, background): “It achieves state-of-the-art performance on automatic speech recognition (ASR) (Park et al., 2019; Synnaeve et al., 2019) and leads to the recent resurgence of speech-to-text translation (ST) research (Duong et al., 2016; B´erard et al., 2016).”
The zero resource speech challenge 2017 (2017, background): “In this case, a realistic scenario is to process raw speech and its translation in a well-resourced language [6, 7, 8].”
UWSpeech: Speech to Speech Translation for Unwritten Languages (2020, background): “vious works (Muthukumar and Black 2014; Chen et al. 2015; Wilkinson, Zhao, and Black 2016; Adams 2017; Kamper, Livescu, and Goldwater 2017; Dunbar et al. 2017; Eloff et al. 2019; Tjandra et al. 2019; Duong et al. 2016; Salesky, Sperber, and Black 2019) have investigated the conversion between speech”
Listen and Translate: A Proof of Concept for End-to-End Speech-to-Text Translation (2016, neutral): “…from a source phone sequence to a target word sequence (Besacier et al., 2006) or to discover a bilingual lexicon from parallel phone-word sequences (Godard et al., 2016), the only attempt to translate directly a source speech signal into target language text is that of Duong et al. (2016).”
Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio (2019, background): “Cross-lingual translation research has focused on text-to-text translation [21, 22] as well as speechto-text from one language to another [23, 24, 25].”
Speech Translation and the End-to-End Promise: Taking Stock of Where We Are (2020, background): “Duong et al. (2016); Berard et al. (2016); Bansal et al. (2018) explore direct ST models that translate speech without using explicitly generated intermediate ASR output.”
The Multilingual TEDx Corpus for Speech Recognition and Translation (2021, background): “End-to-end models for ST, first explored [45, 46, 47] shortly after similar models achieved success for machine translation, have rapidly become the predominantly researched methodology within the field.”
Learning Shared Semantic Space for Speech-to-Text Translation (2021, unknown): “As a result, there have been a series of recent attempts on end-to-end speech-to-text translation (Liu et al., 2019, 2018; Weiss et al., 2017; B´erard et al., 2018; Duong et al., 2016; Jia et al., 2019; Dong et al., 2021b; Wang et al., 2020b).”
Sequence-to-Sequence Models Can Directly Translate Foreign Speech (2017, methodology): “In [23] seq2seq models were used to align speech with translated text, but not to directly predict the translations.”
Vision as an Interlingua: Learning Multilingual Semantic Embeddings of Untranscribed Speech (2018, background): “Recent work has moved beyond that paradigm by implementing translation between speech audio in the source language and written text in the target language [36, 37, 38].”
Data Augmentation for Pipeline-Based Speech Translation (2020, background): “Speech translation systems are either end-to-end [1,2,3] or pipeline-based where automatic speech recognition (ASR) and machine translation (MT) systems work sequentially [4,5,6].”
GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, methodology): “With the success of attention-based models for speech and text-related tasks, a typical and effective baseline model for ST is speech-transformer [2, 3, 4, 5], which has much of the same model structure as the commonly used MT model Transformer [6], except for the pre-processing down-sampling module”
End-to-End Automatic Speech Translation of Audiobooks (2018, background): “One attempt to translate directly a source speech signal into target language text is that of [1].”
Self-Supervised Representations Improve End-to-End Speech Translation (2020, background): “Recently, there has been much interest in end-to-end speech translation (ST) models [1, 2, 3, 4, 5, 6, 7], which, compared to traditional cascaded models, are simpler and computationally more efficient, can preserve more acoustic information and can avoid propagating errors from the speech recogniti”
MAM: Masked Acoustic Modeling for End-to-End Speech-to-Text Translation (2020, unknown): “standard orthography, making phonetic transcription impossible (Duong et al., 2016).”
Improving Cross-Lingual Transfer Learning for End-to-End Speech Recognition with Speech Translation (2020, background): “The recent surge of end-to-end speech-to-text translation (ST) studies [7, 8, 9, 10, 11] is also due to the application of attention-based encoder-decoder models.”
Neural Machine Translation: A Review (2019, unknown): “, 2016), cross-lingual word-to-phone alignment (Duong et al., 2016), bioinformatics (Sønderby et al.”
Semantic Speech Retrieval With a Visually Grounded Model of Untranscribed Speech (2017, background): “…models that, instead of using exact transcriptions, can learn from weaker forms of supervision, e.g., known word pairs (Syn-naeve et al., 2014a; Settle et al., 2017), translation text (Duong et al., 2016; Bansal et al., 2017; Weiss et al., 2017), or unordered word labels (Palaz et al., 2016).”