Acoustically Grounded Word Embeddings for Improved Acoustics-to-word Speech Recognition

By Shane Settle, Kartik Audhkhasi, Karen Livescu, Michael A. Picheny. 2019

35 citations recorded by the source corpus.

Direct acoustics-to-word (A2W) systems for end-to-end automatic speech recognition are simpler to train, and more efficient to decode with, than sub-word systems. However, A2W systems can have difficulties at training time when data is limited, and at decoding time when recognizing words outside the training vocabulary. To address these shortcomings, we investigate the use of recently proposed acoustic and acoustically grounded word embedding techniques in A2W systems. The idea is based on treating the final pre-softmax weight matrix of an AWE recognizer as a matrix of word embedding vectors, and using an externally trained set of word embeddings to improve the quality of this matrix. In particular we introduce two ideas: (1) Enforcing similarity at training time between the external embeddings and the recognizer weights, and (2) using the word embeddings at test time for predicting out-of-vocabulary words. Our word embedding model is acoustically grounded, that is it is learned jointly with acoustic embeddings so as to encode the words’ acoustic-phonetic content; and it is parametric, so that it can embed any arbitrary (potentially out-of-vocabulary) sequence of characters. We find that both techniques improve the performance of an A2W recognizer on conversational telephone speech.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “Although there are few works that allow injecting any out-ofvocabulary word into the vocabulary of acoustic-to-word E2E models during inference via subword-to-word embedding [45, 247], this problem is far from being solved.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “In addition to the approaches discussed above, several other works on speech representation learning take inspiration from the AE framework [86], [87], [88], [89], [90], [91].”
Effectiveness of Self-Supervised Pre-Training for ASR (2020, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], in addition to semi- and weakly-supervised learning techniques to reduce th”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, methodology): “For the third type of analysis we use mel spectrogram features, acoustically grounded word embeddings [11] and GloVe word embeddings [12] as ways to assess the local acoustic, word-level acoustic and word meaning information encoded in the W2V2 representations respectively.”