Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder

By Yu-An Chung, Chao-Chung Wu, Chia-Hao Shen, Hung-yi Lee, Lin-Shan Lee. 2016

194 citations recorded by the source corpus.

The vector representations of fixed dimensionality for words (in text) offered by Word2Vec have been shown to be very useful in many application scenarios, in particular due to the semantic information they carry. This paper proposes a parallel version, the Audio Word2Vec. It offers the vector representations of fixed dimensionality for variable-length audio segments. These vector representations are shown to describe the sequential phonetic structures of the audio segments to a good degree, with very attractive real world applications such as query-by-example Spoken Term Detection (STD). In this STD application, the proposed approach significantly outperformed the conventional Dynamic Time Warping (DTW) based approaches at significantly lower computation requirements. We propose unsupervised learning of Audio Word2Vec from audio data without human annotation using Sequence-to-sequence Audoencoder (SA). SA consists of two RNNs equipped with Long Short-Term Memory (LSTM) units: the first RNN (encoder) maps the input audio sequence into a vector representation of fixed dimensionality, and the second RNN (decoder) maps the representation back to the input audio sequence. The two RNNs are jointly trained by minimizing the reconstruction error. Denoising Sequence-to-sequence Autoencoder (DSA) is furthered proposed offering more robust learning.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “There also exists some work on unsupervised speech representation learning [11, 12, 13, 14, 15, 16].”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, background): “…framework does not constrain the input or target sequences, and has been successfully applied to tasks such as machine translation [21, 22], video caption generation [23], abstract meaning representation parsing and generation [24], and acoustic word embeddings acquisition [11].”
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders (2019, methodology): “Unsupervised speech representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10] is effective in extracting high-level properties from speech.”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, background): “Although there is work that explores the concept of learning vector representations from speech [18, 19, 20, 21, 22, 23], they are primarily based on acoustic-phonetic similarity, and aim to represent the way a word sounds rather than its meaning.”
Audio Word2vec: Sequence-to-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation (2019, methodology): “Both SA [10] and SSAE [26] have been proposed in previous conference papers; SA’s language transfer ability has also”
Multilingual bottle-neck feature learning from untranscribed speech (2017, background): “In addition to the above frame-wise features, segmentwise features [18, 19, 20] which are represented by a fixed dimensional vector have been proposed recently, and they are usually associated with the discovery of word-like units.”
Effectiveness of Self-Supervised Pre-Training for ASR (2020, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], in addition to semi- and weakly-supervised learning techniques to reduce th”
Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion (2019, background): “Embedding audio signals into latent representations has been a well studied practice [4][5][6][7][8][9][10].”
Learning Word Embeddings from Speech (2017, methodology): “…of challenging tasks such as machine translation [Sutskever et al., 2014, Cho et al., 2014], video caption generation [Venugopalan et al., 2015], abstract meaning representation (AMR) parsing and generation [Konstas et al., 2017], and acquisition of acoustic word embeddings [Chung et al., 2016].”
Self-supervised audio representation learning for mobile devices (2019, background): “More recently, different autoencoder architectures have been explored, e.g., denoising [43], convolutional LSTM autoencoders [26] and sequence-to-sequence autoencoders [7].”
Deep Learning Techniques for Music Generation - A Survey (2017, methodology): “Reproduced from [23] with the permission of the authors”
Gate Activation Signal Analysis for Gated Recurrent Neural Networks and its Correlation with Phoneme Boundaries (2017, methodology): “For problems related to sequential data such as audio, video and text, significant improvements have been achieved by Gated Recurrent Neural Networks (GRNN), in which the hidden neurons form a directed cycle suitable for processing sequential data [4][5][6][7].”
SLAM: A Unified Encoder for Speech and Language Modeling via Speech-Text Joint Pre-Training (2021, methodology): “Chung et al. (2016) follows the word2vec approach to learn vector representations of variable-length audio segments.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, methodology): “However, we find that the core ideas underlying the recent successes in learning local representation models have also been used for global representation learning; masking (Chung et al. 2016), context prediction (Chung and Glass 2018), and contrastive training (Milde and Biemann 2018) have been app”
Segmental Audio Word2Vec: Representing Utterances as Sequences of Vectors with Applications in Spoken Term Detection (2018, background): “In speech signal processing, it has been shown that audio Word2Vec transforming spoken words into vectors of fixed dimensionality[3][4] is also useful for example in spoken term detection or data augmentation[5][6], because those vectors carry phonetic structure for the spoken words.”
Unspeech: Unsupervised Speech Context Embeddings (2018, background): “[11] proposed audio word2vec, based on sequence-to-sequence autoencoders trained on a dictionary of isolated spoken words.”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, methodology): “Holzenberger et al. (2018); Chung et al. (2016) introduced an auto-encoder approach based on recurrent neural networks to extract a ﬁxed-size vector between the encoder and the decoder.”