Multi-view Recurrent Neural Acoustic Word Embeddings

By Wanjia He, Weiran Wang, Karen Livescu. 2016

93 citations recorded by the source corpus.

Recent work has begun exploring neural acoustic word embeddings---fixed-dimensional vector representations of arbitrary-length speech segments corresponding to words. Such embeddings are applicable to speech retrieval and recognition tasks, where reasoning about whole words may make it possible to avoid ambiguous sub-word representations. The main idea is to map acoustic sequences to fixed-dimensional vectors such that examples of the same word are mapped to similar vectors, while different-word examples are mapped to very different vectors. In this work we take a multi-view approach to learning acoustic word embeddings, in which we jointly learn to embed acoustic sequences and their corresponding character sequences. We use deep bidirectional LSTM embedding models and multi-view contrastive losses. We study the effect of different loss variants, including fixed-margin and cost-sensitive losses. Our acoustic word embeddings improve over previous approaches for the task of word discrimination. We also present results on other tasks that are enabled by the multi-view approach, including cross-view word discrimination and word similarity.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Speech2Vec: A Sequence-to-Sequence Framework for Learning Word Embeddings from Speech (2018, background): “Researchers have also explored the concept of learning vector representations from speech [9, 10, 11, 12, 13, 14].”
Unsupervised Cross-Modal Alignment of Speech and Text Embedding Spaces (2018, background): “speech [18, 19, 20, 21, 22, 23], they are primarily based on acoustic-phonetic similarity, and aim to represent the way a word sounds rather than its meaning.”
An embedded segmental K-means model for unsupervised segmentation and clustering of speech (2017, background): “There is a growing focus on such acoustic word embedding methods [11, 32–38], since they make it possible to easily and efficiently compare variableduration speech segments in a fixed-dimensional space.”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “The representation loss is a contrastive loss, similar to ones used in prior work on multi-view representation learning [2,17, 18], computed by sampling a fixed number of negative examples within a mini-batch (sizeB) corresponding to both images and utterances:”
Learning Word Embeddings from Speech (2017, background): “Previous research has explored the concept of learning vector representations from speech [He et al., 2017, Kamper et al., 2016, Chung et al., 2016, Settle and Livescu, 2016, Bengio and Heigold, 2014, Levin et al., 2013].”
Unsupervised End-to-End Learning of Discrete Linguistic Units for Voice Conversion (2019, background): “Embedding audio signals into latent representations has been a well studied practice [4][5][6][7][8][9][10].”
Learning from Multiview Correlations in Open-domain Videos (2018, background): “Representations can also be learned independently of a specific task, for instance with a hinge loss, where a neural network is trained to distinguish matched from mismatched pairs across or within modalities [14].”
Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition (2019, methodology): “In particular, our embedding approach closely follows that of [12], which jointly learns an acoustic embedding function—mapping an acoustic signal to a ﬁxed-dimensional vector— and an acoustically grounded textual embedding function—mapping a character sequence to a vector.”
Extracting bottleneck features and word-like pairs from untranscribed speech for feature representation (2017, background): “Our future work will investigate the use of more sophisticated neural networks [24, 25] in pairwise learning to improve our proposed features.”