Towards Visually Grounded Sub-word Speech Unit Discovery

By David F. Harwath, James R. Glass. 2019

36 citations recorded by the source corpus.

In this paper, we investigate the manner in which interpretable sub-word speech units emerge within a convolutional neural network model trained to associate raw speech waveforms with semantically related natural image scenes. We show how diphone boundaries can be superficially extracted from the activation patterns of intermediate layers of the model, suggesting that the model may be leveraging these events for the purpose of word recognition. We present a series of experiments investigating the information encoded by these events.

Subfields: Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Speech & Audio

Citations in the corpus

Unsupervised Speech Recognition (2021, methodology): “Method Precision Recall F1 DAVEnet + peak detection (Harwath and Glass, 2019) .”
Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “Previous analyses reveal that ResDAVEnet-like models learn linguistic abstractions at different levels, including words (Harwath & Glass, 2017) and robust phonetic features (Harwath & Glass, 2019; Hsu et al., 2019).”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “…Harwath, Hsu, and Glass 2020), cross-modality alignment (Harwath et al. 2019; Wang et al. 2021; Khorrami and R ¨ as ¨ anen 2021), speech segmentation (Harwath and Glass 2019), speech generation (Hsu et al. 2021b), and learning multilingual speech representations (Harwath, Chuang, and Glass…”
On the Contributions of Visual and Textual Supervision in Low-resource Semantic Speech Retrieval (2019, unknown): “Such work has developed approaches for training models for tasks such as cross-modal retrieval [2,3]; unsupervised learning of word-like and phone-like units [7–10]; and retrieval tasks like keyword search, query-by-example search, and semantic search [6,11,12].”
Large-Scale Representation Learning from Visually Grounded Untranscribed Speech (2019, background): “…images to audio captions that describe them (Figure 1), building on previous research into learning from visually grounded, untranscribed speech (Harwath and Glass, 2015; Sun et al., 2016; Harwath et al., 2016; Chrupała et al., 2017; Kamper et al., 2017b; Chrupała, 2019; Harwath and Glass, 2019).”
Text-Free Image-to-Speech Synthesis Using Learned Segmental Units (2020, background): “…2019), as well as how word-like and subword-like linguistic units can be made to emerge within these models (Har-wath and Glass, 2017; Harwath et al., 2019; Drexler and Glass, 2017; Alishahi et al., 2017; Harwath et al., 2019; Harwath and Glass, 2019; Havard et al., 2019b; Harwath et al., 2020).”
Word Recognition, Competition, and Activation in a Model of Visually Grounded Speech (2019, background): “Recently, Harwath et al. (2018) showed that CNN-based models could reliably map word-like units to their visual referents, and Harwath and Glass (2019) showed such networks were sensitive to diphone transitions and that these were useful for the purpose of word recognition.”