Discriminative acoustic word embeddings: Tecurrent neural network-based approaches

By Shane Settle, Karen Livescu. 2016

92 citations recorded by the source corpus.

Acoustic word embeddings — fixed-dimensional vector representations of variable-length spoken word segments — have begun to be considered for tasks such as speech recognition and query-by-example search. Such embeddings can be learned discriminatively so that they are similar for speech segments corresponding to the same word, while being dissimilar for segments corresponding to different words. Recent work has found that acoustic word embeddings can outperform dynamic time warping on query-by-example search and related word discrimination tasks. However, the space of embedding models and training approaches is still relatively unexplored. In this paper we present new discriminative embedding models based on recurrent neural networks (RNNs). We consider training losses that have been successful in prior work, in particular a cross entropy loss for word classification and a contrastive loss that explicitly aims to separate same-word and different-word pairs in a “Siamese network” training setting. We find that both classifier-based and Siamese RNN embeddings improve over previously reported results on a word discrimination task, with Siamese RNNs outperforming classification models. In addition, we present analyses of the learned embeddings and the effects of variables such as dimensionality and network structure.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “However, there has been evidence [2] and theories [3] supporting the idea that low contrastive loss implies the existence of a linear classifier with low error.”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, background): “in text, and contain semantic information about the word which are useful for many NLP tasks [4, 5, 6, 7, 8]. Researchers have also explored the concept of learning vector representations from speech [9, 10, 11, 12, 13, 14]. These approaches are based on notions of acoustic-phonetic (rather than sem”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, background): “Although there is work that explores the concept of learning vector representations from speech [18, 19, 20, 21, 22, 23], they are primarily based on acoustic-phonetic similarity, and aim to represent the way a word sounds rather than its meaning.”
An embedded segmental K-means model for unsupervised segmentation and clustering of speech (2017, background): “There is a growing focus on such acoustic word embedding methods [11, 32–38], since they make it possible to easily and efficiently compare variableduration speech segments in a fixed-dimensional space.”
Audio Word2vec: Sequence-to-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation (2019, unknown): “representation of the input segment [37], [45].”
Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition (2019, background): “Several approaches have been developed for learning acoustic word embedding models—functions mapping arbitrary-duration spoken word signals to fixed-dimensional vectors—so as to encode either phonetic [10, 11, 20–24] or semantic [25] information, or both [26].”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, background): “For example, some recent work uses convolutional or recurrent neural networks learned by optimizing a contrastive loss using a set of known same-word pairs [26, 28].”
Learning Word Embeddings from Speech (2017, background): “Previous research has explored the concept of learning vector representations from speech [He et al., 2017, Kamper et al., 2016, Chung et al., 2016, Settle and Livescu, 2016, Bengio and Heigold, 2014, Levin et al., 2013].”
Multimodal and Multilingual Embeddings for Large-Scale Speech Mining (2021, background): “Some other works (Settle and Livescu, 2016; Riad et al., 2018; Thiolliere et al., 2015) train Siamese networks with a contrastive loss to build a ﬁxed-size representation.”
Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings (2017, methodology): “e. Here d cos(x 1;x 2) = (1 cos(x 1;x 2)) is the cosine distance between vectors x 1 and x 2. Rather than sampling a single negative example as in [12], or keeping track of confusion statistics as in [13], we sample a set of kembedded segments Dfrom the whole training set with labels different from ”
Multi-view Recurrent Neural Acoustic Word Embeddings (2016, methodology): “We use the same experimental setup and data as in Kamper et al. (2015, 2016); Settle & Livescu (2016).”