Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection

By Yu-Hsuan Wang, Hung-yi Lee, Lin-Shan Lee. 2018

55 citations recorded by the source corpus.

While Word2Vec represents words (in text) as vectors carrying semantic information, audio Word2Vec was shown to be able to represent signal segments of spoken words as vectors carrying phonetic structure information. Audio Word2Vec can be trained in an unsupervised way from an unlabeled corpus, except the word boundaries are needed. In this paper, we extend audio Word2Vec from word-level to utterance-level by proposing a new segmental audio Word2Vec, in which unsupervised spoken word boundary segmentation and audio Word2Vec are jointly learned and mutually enhanced, so an utterance can be directly represented as a sequence of vectors carrying phonetic structure information. This is achieved by a segmental sequence-to-sequence autoencoder (SSAE), in which a segmentation gate trained with reinforcement learning is inserted in the encoder. Experiments on English, Czech, French and German show very good performance in both unsupervised spoken word segmentation and spoken term detection applications (significantly better than frame-based DTW).

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Towards Unsupervised Speech Recognition and Synthesis with Quantized Speech Representation Learning (2019, background): “audio word vectors) were also developed with the boundaries for the linguistic units needed as ground truth [6] or automatically detected [7] but at the cost of the quality of the representations de-”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, methodology): “In this respect our approach is similar to those in recent embedding-based QbE work [14–17], and also some embedding-based spoken term detection work [18].”
Utilizing Self-supervised Representations for MOS Prediction (2021, methodology): “Figure 2 explains our framework, which consists of segmental embeddings [25, 26, 27, 28] of pretrained model (Sec.”
SpeechBERT: An Audio-and-Text Jointly Learned Language Model for End-to-End Spoken Question Answering (2019, background): “It was proposed [18] to learn to jointly segment the audio words out of utterance and extract the embeddings.”
Audio Word2vec: Sequence-to-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation (2019, methodology): “We propose the segmental sequence-to-sequence autoencoder (SSAE) [26], a new model to jointly train the segmenter while extracting the representation.”
Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings (2018, methodology): “Many available approaches can be used here for this purpose, but below we assume the recently developed Segmental Audio Word2Vec is used [28].”
Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data (2018, methodology): “Segmental Audio Word2Vec was further proposed to jointly segment an utterance into a sequence of spoken words and transform them into a sequence of vectors [8].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “[264] Audio word2vec [265] Oracle Phoneme GAN TIMIT (PER): 63.”