Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling
By Puyuan Peng, David F. Harwath. 2022
30 citations recorded by the source corpus.
In this paper, we describe our submissions to the ZeroSpeech 2021 Challenge and SUPERB benchmark. Our submissions are based on the recently proposed FaST-VGS model, which is a Transformer-based model that learns to associate raw speech waveforms with semantically related images, all without the use of any transcriptions of the speech. Additionally, we introduce a novel extension of this model, FaST-VGS+, which is learned in a multi-task fashion with a masked language modeling objective in addition to the visual grounding objective. On ZeroSpeech 2021, we show that our models perform competitively on the ABX task, outperform all other concurrent submissions on the Syntactic and Semantic tasks, and nearly match the best system on the Lexical task. On the SUPERB benchmark, we show that our models also achieve strong performance, in some cases even outperforming the popular wav2vec2.0 model.
Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Neil Zeghidour
- Zalán Borsos
- Marco Tagliasacchi
- Matthew Sharifi
- Damien Vincent
- Evgeny Kharitonov
- Olivier Pietquin
- Olivier Teboul
In Computer Vision and Pattern Recognition
- Neil Zeghidour
- Olivier Pietquin
- Zalán Borsos
- Evgeny Kharitonov
- Marco Tagliasacchi
- Matthew Sharifi
- David Grangier
- Dominik Roblek
Citations in the corpus
- AudioLM: A Language Modeling Approach to Audio Generation (2022, methodology): “[59] which is based on a RoBERTA [60] model trained on top of speech representations obtained with visual grounding.”
- Self-Supervised Speech Representation Learning: A Review (2022, unknown): “ZeroSpeech and SUPERB benchmark tasks [180].”