Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech

By Yu-An Chung, James R. Glass. 2018

189 citations recorded by the source corpus.

In this paper, we propose a novel deep neural network architecture, Speech2Vec, for learning fixed-length vector representations of audio segments excised from a speech corpus, where the vectors contain semantic information pertaining to the underlying spoken words, and are close to other vectors in the embedding space if their corresponding underlying spoken words are semantically similar. The proposed model can be viewed as a speech version of Word2Vec. Its design is based on a RNN Encoder-Decoder framework, and borrows the methodology of skipgrams or continuous bag-of-words for training. Learning word embeddings directly from speech enables Speech2Vec to make use of the semantic information carried by speech that does not exist in plain text. The learned word embeddings are evaluated and analyzed on 13 widely used word similarity benchmarks, and outperform word embeddings learned by Word2Vec from the transcriptions.

Subfields: Artificial Intelligence, Natural Language Processing, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Speech & Audio

In Signal Processing

Citations in the corpus

Unsupervised Speech Recognition (2021, background): “, 2020) and self-supervised learning (van den Oord et al., 2018; Chung and Glass, 2018; Chung et al., 2019b; Baevski et al., 2020c).”
Libri-Light: A Benchmark for ASR with Limited or No Supervision (2019, neutral): “Yet another approach might apply language modeling techniques directly on unlabelled audio to improve the representations before fine-tuning them [26, 27].”
vq-wav2vec: Self-Supervised Learning of Discrete Speech Representations (2019, background): “Another line of research is to learn continuous speech representations in a self-supervised way via predicting context information (Chung & Glass, 2018; van den Oord et al., 2018; Schneider et al., 2019).”
An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “There also exists some work on unsupervised speech representation learning [11, 12, 13, 14, 15, 16].”
Unsupervised Speech Representation Learning Using WaveNet Autoencoders (2019, methodology): “[76] replace the segmental autoencoder with a model that instead predicts a nearby speech”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, background): “…speech representation learning has also been investigated and shown promising results, beneﬁting from richly learned representations (Chung and Glass, 2018; Chuang et al., 2020; Song et al., 2019; Baevski et al., 2020; Wang et al., 2021; Hsu et al., 2021; Chung et al., 2021a), such…”
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition (2019, background): “More recently, acoustic representation learning has drawn increasing attention [19, 20, 21, 22, 23, 24] in speech processing.”
Generative Pre-Training for Speech with Autoregressive Predictive Coding (2019, background): “However, most existing representation learning objectives [6, 7, 8, 9, 10] are designed to remove certain variabilities in speech (such as noise or speaker, depending on their design) and thus risk discarding information that could be useful for unknown downstream tasks.”
Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders (2019, methodology): “Unsupervised speech representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10] is effective in extracting high-level properties from speech.”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, methodology): “Recently, the Speech2Vec [24] model was developed to be capable of representing audio segments excised from a speech corpus as fixed dimensional vectors that contain semantic information of the underlying spoken words.”
Self-supervised audio representation learning for mobile devices (2019, background): “Recently, a similar approach has been proposed to map speech to ﬁxed-dimensional embeddings [6, 9].”
Effectiveness of Self-Supervised Pre-Training for ASR (2020, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], in addition to semi- and weakly-supervised learning techniques to reduce th”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, methodology): “2016), context prediction (Chung and Glass 2018), and contrastive training (Milde and Biemann 2018) have been applied in both settings.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “For example, searching within a corpus of recorded speech for segments that match a given (written or spoken) query can be seen as finding segments whose representations are most similar to that of the query [118], [192], [193], [194]; word embeddings can be defined by pooling representations of ins”