Visually grounded cross-lingual keyword spotting in speech

By Herman Kamper, Michael Roth. 2018

34 citations recorded by the source corpus.

Recent work considered how images paired with speech can be used as supervision for building speech systems when transcriptions are not available. We ask whether visual grounding can be used for cross-lingual keyword spotting: given a text keyword in one language, the task is to retrieve spoken utterances containing that keyword in another language. This could enable searching through speech in a low-resource language using text queries in a high-resource language. As a proof-of-concept, we use English speech with German queries: we use a German visual tagger to add keyword labels to each training image, and then train a neural network to map English speech to German keywords. Without seeing parallel speech-transcriptions or translations, the model achieves a precision at ten of 58%. We show that most erroneous retrievals contain equivalent or semantically relevant keywords; excluding these would improve P@10 to 91%.

Subfields: Artificial Intelligence, Multimodal Learning, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Multimodal Learning

In Speech & Audio

Citations in the corpus

Learning Hierarchical Discrete Linguistic Units from Visually-Grounded Speech (2019, background): “…Kamper et al., 2019b;a; Surı́s et al., 2019; Ilharco et al., 2019; Eloff et al., 2019a), application to multilingual settings (Harwath et al., 2018a; Kamper & Roth, 2017; Azuh et al., 2019; Havard et al., 2019a), analysis of the linguistic abstractions, such as words and phones, which are…”
Towards Bilingual Lexicon Discovery From Visually Grounded Speech Audio (2019, background): “A similar line of work was presented in [27], which explored cross-lingual keyword spotting using a visual tagging system.”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “Starting from the work of Synnaeve, Versteegh, and Dupoux (2014); Har-wath and Glass (2015), researchers have studied the ability of models to learn to recognize the structure of spoken language, such as words and sub-word units, by training the models to associate speech waveforms with contextually”
Semantic Query-by-example Speech Search Using Visual Grounding (2019, background): “One form of weak supervision that has seen recent success is visual grounding: the use of images paired with speech data [4– 8].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “be used for cross-lingual retrieval, by considering the visual signal as an “interlingua” [169]–[171].”