Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech

By David F. Harwath, Wei-Ning Hsu, James R. Glass. 2019

88 citations recorded by the source corpus.

In this paper, we present a method for learning discrete linguistic units by incorporating vector quantization layers into neural models of visually grounded speech. We show that our method is capable of capturing both word-level and sub-word units, depending on how it is configured. What differentiates this paper from prior work on speech unit learning is the choice of training objective. Rather than using a reconstruction-based loss, we use a discriminative, multimodal grounding objective which forces the learned units to be useful for semantic image retrieval. We evaluate the sub-word units on the ZeroSpeech 2019 challenge, achieving a 27.3% reduction in ABX error rate over the top-performing submission, while keeping the bitrate approximately the same. We also present experiments demonstrating the noise robustness of these units. Finally, we show that a model with multiple quantizers can simultaneously learn phone-like detectors at a lower layer and word-like detectors at a higher layer. We show that these detectors are highly accurate, discovering 279 words with an F1 score of greater than 0.5.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Natural Language Processing, Multimodal Learning, Representation & Self-Supervised Learning, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Natural Language Processing

In Multimodal Learning

In Representation & Self-Supervised Learning

In Signal Processing

Citations in the corpus

Experience Grounds Language (2020, background): “, 2019) to facilitate discovery of linguistic concepts (Harwath et al., 2020).”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, methodology): “As part of training, we learn discrete speech units [53, 32, 7, 18] via a gumbel softmax [24, 5] to represent the latent representations in the contrastive task (Figure 1) which we find to be more effective than non-quantized targets.”
Unsupervised Speech Recognition (2021, background): “, 2020) as well as work on combining modalities such as speech and vision (Harwath et al., 2020).”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, background): “Self-supervised learning of speech representations has received a lot attention [1, 2, 3, 4, 5] and has been demonstrated to work well both in low- and high-resource labeled data settings for automatic speech recognition (ASR) [6].”
Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, background): “In speech processing, most work in this area has focused on monolingual unsupervised representation learning [18, 19, 20, 21, 22, 23, 24, 25].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “One motivation to apply vector quantization (VQ) is that enforcing quantization can lead to better linguistic unit discovery [28, 29] due to the discrete nature of phonetic units.”
Vector-Quantized Autoregressive Predictive Coding (2020, background): “Recent studies on VQ for representation learning, mostly motivated by the discrete nature of phonetic units, attempt to show that enforcing the quantization leads to a better representation for acoustic unit discovery [16, 17] and automatic speech recognition [7, 18].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “other contextual modality), such models often learn multiple levels of linguistic representations from the shallowest to the deepest model layers [181], [182], [183].”
Vector-quantized neural networks for acoustic unit discovery in the ZeroSpeech 2020 challenge (2020, methodology): “Finally, it is interesting to note that our models (trained exclusively on unlabelled speech) achieve comparable ABX scores to the visually grounded VQ model of [42] – which is trained on paired images and unlabelled spoken captions.”
fairseq Sˆ2: A Scalable and Integrable Speech Synthesis Toolkit (2021, background): “Beyond text-based systems, there is also an emerging research topic that explores the use of units discovered from self-supervised speech representation learning (Oord et al., 2017; Baevski et al., 2019; Harwath et al., 2019; Hsu et al., 2021)”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “…2019; Wang and Hasegawa-Johnson 2020; Olaleye and Kamper 2021), hierarchical linguistic unit analysis (Chrupała, Gelderloos, and Alishahi 2017; Harwath, Hsu, and Glass 2020), cross-modality alignment (Harwath et al. 2019; Wang et al. 2021; Khorrami and R ¨ as ¨ anen 2021), speech…”
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning (2020, background): “Although, generative models are theoretically elegant and informed by theories of cognition, most recent success in speech representation learning has come from self-supervised learning algorithms such as Wav2Vec [8], Problem Agnostic Speech Encoding (PASE) [9], Autoregressive Predictive Coding (APC”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, methodology): “We follow the data splits used in (Harwath et al., 2020) for Places, and (Karpathy and Fei-Fei, 2015) for Flickr8k and SpokenCOCO (commonly known as the “Karpathy split”).”