Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces

By Yu-An Chung, Wei-Hung Weng, Schrasing Tong, James R. Glass. 2018

108 citations recorded by the source corpus.

Recent research has shown that word embedding spaces learned from text corpora of different languages can be aligned without any parallel data supervision. Inspired by the success in unsupervised cross-lingual word embeddings, in this paper we target learning a cross-modal alignment between the embedding spaces of speech and text learned from corpora of their respective modalities in an unsupervised fashion. The proposed framework learns the individual speech and text embedding spaces, and attempts to align the two spaces via adversarial training, followed by a refinement procedure. We show how our framework could be used to perform spoken word classification and translation, and the results on these two tasks demonstrate that the performance of our unsupervised alignment approach is comparable to its supervised counterpart. Our framework is especially useful for developing automatic speech recognition (ASR) and speech-to-text translation systems for low- or zero-resource languages, which have little parallel audio-text data for training modern supervised ASR and speech-to-text translation models, but account for the majority of the languages spoken across the world.

Subfields: Artificial Intelligence, Multimodal Learning, Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Representation & Self-Supervised Learning

In Speech & Audio

Citations in the corpus

Unsupervised Speech Recognition (2021, background): “Segmentation has been shown to be crucial in prior work (Chung et al., 2018) since the right boundaries in the input representations make it easier to predict the output sequence.”
Libri-Light: A Benchmark for ASR with Limited or No Supervision (2019, positive): “For g c , we use ﬁve convolutional layers with strides [5, 4, 2, 2, 2], ﬁlter-sizes [10, 8, 4, 4, 4] and 256 hidden units with ReLU activations.”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “There has been work on unsupervised learning for speech but the resulting representations have not been applied to improve supervised speech recognition (Synnaeve & Dupoux, 2016b; Kamper et al., 2017; Chung et al., 2018; Chen et al., 2018; Chorowski et al., 2019).”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, methodology): “of word embeddings discovered on separate corpora [77] the approach yielded an ASR system trained on unpaired speech and text data [78].”
Performance-Efficiency Trade-Offs in Unsupervised Pre-Training for Speech Recognition (2021, background): “Besides ASR-focused works, there is significant interest in learning representations for other speech tasks (Synnaeve and Dupoux, 2016; Chung et al., 2018; Chuang et al., 2019; Song et al., 2019), music (Yang et al., 2021; Zhao and Guo, 2021), and general audio (Saeed et al., 2021; Gong et al.,…”
Speech Recognition (2018, unknown): “There has been work on unsupervised learning for speech but the resulting representations have not been applied to improve supervised speech recognition (Synnaeve & Dupoux, 2016b; Kamper et al., 2017; Chung et al., 2018; Chen et al., 2018; Chorowski et al., 2019).”
Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching (2018, background): “More recently, Chung et al. (2018) show that unsupervised spoken word classification is possible by using adversarial cross-modal alignments similar to that in uNMT systems.”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, unknown): “Exceptions include [18,31], which use no annotations.”
SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering (2019, background): “Other efforts then tried to align the audio word embeddings with text word embeddings [19, 20].”
Deep Partial Multi-View Learning (2020, methodology): “With adversarial training, the embedding spaces of two individual views are learned and aligned simultaneously [30].”
St-Bert: Cross-Modal Language Model Pre-Training for End-to-End Spoken Language Understanding (2020, background): “[17] learn audio segment representations and word representations individually and aligns their spaces via adversarial training.”
Self-supervised audio representation learning for mobile devices (2019, background): “Learning multimodal representations : Several papers have recently investigated learning audio representations exploiting the correlation with other modalities, e.g., text [8], images [32] and videos [31, 16, 1, 22, 10].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “[286], [288] use adversarial training to map the word-level speech embedding”
Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data (2018, methodology): “However, semantic information has to be learned from the context out of extremely large amount of data, such as hundreds of hours of speech data [11, 13], while rarely used words still have relatively poor embeddings for lack of contextual information.”
Completely Unsupervised Speech Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Models (2019, methodology): “But there still exists a wide gap between all these works and unsupervised ASR.”
A Cookbook of Self-Supervised Learning (2023, background): “Multi-modal models, on the other hand, can consider a soundbite and its text [Sermanet et al., 2018, Chung et al., 2018] or some frames of a video and the corresponding sound clip [Zhao et al., 2018, Alwassel et al., 2020] as different views to be used as positive pairs for contrastive learning.”