Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings

By Da-Rong Liu, Kuan-Yu Chen, Hung-yi Lee, Lin-Shan Lee. 2018

51 citations recorded by the source corpus.

Unsupervised discovery of acoustic tokens from audio corpora without annotation and learning vector representations for these tokens have been widely studied. Although these techniques have been shown successful in some applications such as query-by-example Spoken Term Detection (STD), the lack of mapping relationships between these discovered tokens and real phonemes have limited the down-stream applications. This paper represents probably the first attempt towards the goal of completely unsupervised phoneme recognition, or mapping audio signals to phoneme sequences without phoneme-labeled audio data. The basic idea is to cluster the embedded acoustic tokens and learn the mapping between the cluster sequences and the unknown phoneme sequences with a Generative Adversarial Network (GAN). An unsupervised phoneme recognition accuracy of 36% was achieved in the preliminary experiments.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Recognition (2021, methodology): “We use adversarial training to train an unsupervised speech recognition model using the representations of the unlabeled speech audio data and the unlabeled phonemized text data (Liu et al., 2018; Chen et al., 2019).”
Almost Unsupervised Text to Speech and Automatic Speech Recognition (2019, background): “Works focusing on unsupervised ASR (Chen et al., 2018a; Liu et al., 2018) do not leverage additional information from TTS, which is a dual task to ASR and is of great potential to improve the performance of ASR.”
LRSpeech: Extremely Low-Resource Speech Synthesis and Recognition (2020, background): “• In the unsupervised setting, only unpaired speech and text data are leverage to build ASR models [8, 24, 45].”
Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data (2018, background): “Un-supervised phoneme recognition was then achieved to some extent by making the duration of spoken words shorter and mapping the clusters of vector representations to phonemes even without aligned audio and text [9].”
Unsupervised Speech Recognition via Segmental Empirical Output Distribution Matching (2018, methodology): “This is different from the previous work (Liu et al., 2018), which first performs clustering in the speech space and then maps the clusters into output space using adversarial training.”
Completely Unsupervised Speech Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Models (2019, methodology): “This included aligning audio and text embedding spaces for the purpose of unsupervised ASR [18, 19, 20], and our prior work [21] of unsupervised phoneme recognition, with a Generative Adversarial Network (GAN) [17], by clustering audio embeddings into a set of tokens, and learning the mapping relati”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “use a phoneme segmentation module before the generator to segment utterances into phoneme-level segments [283], [290], [293].”