Unspeech: Unsupervised Speech Context Embeddings

By Benjamin Milde, Chris Biemann. 2018

28 citations recorded by the source corpus.

We introduce "Unspeech" embeddings, which are based on unsupervised learning of context feature representations for spoken language. The embeddings were trained on up to 9500 hours of crawled English speech data without transcriptions or speaker information, by using a straightforward learning objective based on context and non-context discrimination with negative sampling. We use a Siamese convolutional neural network architecture to train Unspeech embeddings and evaluate them on speaker comparison, utterance clustering and as a context feature in TDNN-HMM acoustic models trained on TED-LIUM, comparing it to i-vector baselines. Particularly decoding out-of-domain speech data from the recently released Common Voice corpus shows consistent WER reductions. We release our source code and pre-trained Unspeech models under a permissive open source license.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

Citations in the corpus

Unsupervised Pre-Training of Bidirectional Speech Encoders via Masked Reconstruction (2020, background): “Other work considers a variety of downstream supervised tasks, and some focuses explicitly on learning representations that generalize across tasks or across very different domains [6, 7, 18, 19].”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “…used to learn powerful speech representations using weakly or unsupervised objectives (Thiolliere et al., 2015; Kamper et al., 2015; Hsu et al., 2017a;b; Hsu & Glass, 2018; Holzenberger et al., 2018; Milde & Biemann, 2018; van den Oord et al., 2018; Chung et al., 2019; Pascual et al., 2019).”
An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “There also exists some work on unsupervised speech representation learning [11, 12, 13, 14, 15, 16].”
Generative Pre-Training for Speech with Autoregressive Predictive Coding (2019, background): “However, most existing representation learning objectives [6, 7, 8, 9, 10] are designed to remove certain variabilities in speech (such as noise or speaker, depending on their design) and thus risk discarding information that could be useful for unknown downstream tasks.”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, positive): “This idea of learning contextual embedding based on the local neighbors in the sequence have been found useful in the field of learning word embedding [20, 12] and have been extended to speech representation learning before [21, 22, 23, 24, 25].”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “…core ideas underlying the recent successes in learning local representation models have also been used for global representation learning; masking (Chung et al. 2016), context prediction (Chung and Glass 2018), and contrastive training (Milde and Biemann 2018) have been applied in both settings.”
Improved Speech Representations with Multi-Target Autoregressive Predictive Coding (2020, background): “Unsupervised speech representation learning, which aims to learn a function that transforms surface features, such as audio waveforms or spectrograms, to higher-level representations using only unlabeled speech, has received great attention recently (Baevski et al., 2019, 2020; Liu et al., 2020; Son”
Self-Supervised Speech Representation Learning: A Review (2022, unknown): “Unspeech [101] X[t1,t2] X[t0,t1],X[t2,t3]”