Deep convolutional acoustic word embeddings using word-pair side information

By Herman Kamper, Weiran Wang, Karen Livescu. 2015

181 citations recorded by the source corpus.

Recent studies have been revisiting whole words as the basic modelling unit in speech recognition and query applications, instead of phonetic units. Such whole-word segmental systems rely on a function that maps a variable-length speech segment to a vector in a fixed-dimensional space; the resulting acoustic word embeddings need to allow for accurate discrimination between different word types, directly in the embedding space. We compare several old and new approaches in a word discrimination task. Our best approach uses side information in the form of known word pairs to train a Siamese convolutional neural network (CNN): a pair of tied networks that take two speech segments as input and produce their embeddings, trained with a hinge loss that separates same-word pairs and different-word pairs by some margin. A word classifier CNN performs similarly, but requires much stronger supervision. Both types of CNNs yield large improvements over the best previously published results on the word discrimination task.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

An Unsupervised Autoregressive Model for Speech Representation Learning (2019, background): “The autoregressive approach also does not require other types of linguistic constraints, such as phonetic or word boundaries [7].”
Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, background): “Researchers have also explored the concept of learning vector representations from speech [9, 10, 11, 12, 13, 14].”
A segmental framework for fully-unsupervised large-vocabulary speech recognition (2016, methodology): “A more accurate description would be pseudo term, but we use word instead to match usage in earlier work [1, 25, 42].”
Unsupervised Learning of Semantic Audio Representations (2017, positive): “Various forms of deep metric learning [17, 18, 19, 20] have been successfully applied using these speech-specific constraints.”
Audio Word2Vec: Unsupervised Learning of Audio Segment Representations Using Sequence-to-Sequence Autoencoder (2016, background): “Different from the previous work [12, 13, 11], here learning SA does not need any supervision.”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, background): “speech [18, 19, 20, 21, 22, 23], they are primarily based on acoustic-phonetic similarity, and aim to represent the way a word sounds rather than its meaning.”
An embedded segmental K-means model for unsupervised segmentation and clustering of speech (2017, methodology): “Here we instead follow an acoustic word embedding approach [11,32,33]: an embedding function fe is used to map a variable length speech segment to a single embedding vector x ∈ R in a fixed-dimensional space, i.”
Discriminative acoustic word embeddings: Tecurrent neural network-based approaches (2016, methodology): “…subset of the training set that has a sufﬁcient number of segments per word to train a good classiﬁer, and the output dimensionality is equal to the number of words (but see [14] for a study of varying the dimensionality in such a classiﬁer-based embedding model by introducing a bottleneck layer).”
Learning Weakly Supervised Multimodal Phoneme Embeddings (2017, methodology): “In particular, we will use a Siamese DNN architecture which feeds on word-level side information (the fact that two words are the same or different: [5, 6, 7, 8]), which demonstrably can be discovered automatically from continuous speech using spoFigure 1: Multi-task setting.”
Unsupervised Word Segmentation and Lexicon Discovery Using Acoustic Word Embeddings (2016, background): “embedding techniques are the subject of current research [28] and it would be straightforward to replace the current embedding approach with any other (including one that incorporates”
Multilingual bottle-neck feature learning from untranscribed speech (2017, background): “In addition to the above frame-wise features, segmentwise features [18, 19, 20] which are represented by a fixed dimensional vector have been proposed recently, and they are usually associated with the discovery of word-like units.”
Audio Word2vec: Sequence-to-Sequence Autoencoding for Unsupervised Learning of Audio Segmentation and Representation (2019, background): “In [38], the authors train a neural network with side information to obtain embeddings that separate same-”
Visually Grounded Learning of Keyword Prediction from Untranscribed Speech (2017, methodology): “It is a CNN based on [16, 37], consisting of several convolution and max pooling layers (final pooling covering the entire output), followed by fully connected layers.”
ABX-Discriminability Measures and Applications. (Mesures de discriminabilité ABX et applications) (2016, methodology): “11Although this is not our focus in this section, discrimination measures have also been proposed to be of potential interest for purely supervised system design [34, 54, 91] as a rapid evaluation method that does not commit to a particular downstream processing architecture.”
Completely Unsupervised Phoneme Recognition by Adversarially Learning Mapping Relationships from Audio Embeddings (2018, background): “For spoken document retrieval, by representing audio segments as vectors, search can be much more efﬁcient compared to template matching over audio frames [12, 18, 19].”
Completely Unsupervised Speech Recognition By A Generative Adversarial Network Harmonized With Iteratively Refined Hidden Markov Models (2019, background): “Substantial effort was made to learn signal representations directly from speech signals in an unsupervised way, which could be a good step towards unsupervised ASR, and has been shown useful in various tasks, such as speaker identification [3], spoken term detection [4, 5, 6], and spoken document r”
Query-by-Example Search with Discriminative Neural Acoustic Word Embeddings (2017, background): “Rather than sampling a single negative example as in [12], or keeping track of confusion statistics as in [13], we sample a set of k embedded segments D from the whole training set with labels different from Ya and consider only the example embedding, xd ∈ D, that most violates the margin constraint”