Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction

By Bowen Shi, Wei-Ning Hsu, Kushal Lakhotia, Abdel-rahman Mohamed. 2022

490 citations recorded by the source corpus.

Video recordings of speech contain correlated audio and visual information, providing a strong signal for speech representation learning from the speaker's lip movements and the produced sound. We introduce Audio-Visual Hidden Unit BERT (AV-HuBERT), a self-supervised representation learning framework for audio-visual speech, which masks multi-stream video input and predicts automatically discovered and iteratively refined multimodal hidden units. AV-HuBERT learns powerful audio-visual speech representation benefiting both lip-reading and automatic speech recognition. On the largest public lip-reading benchmark LRS3 (433 hours), AV-HuBERT achieves 32.5% WER with only 30 hours of labeled data, outperforming the former state-of-the-art approach (33.6%) trained with a thousand times more transcribed video data (31K hours). The lip-reading WER is further reduced to 26.9% when using all 433 hours of labeled data from LRS3 and combined with self-training. Using our audio-visual representation on the same benchmark for audio-only speech recognition leads to a 40% relative WER reduction over the state-of-the-art performance (1.3% vs 2.3%). Our code and models are available at https://github.com/facebookresearch/av_hubert

Subfields: Computer Vision and Pattern Recognition, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Signal Processing

Citations in the corpus

AudioPaLM: A Large Language Model That Can Speak and Listen (2023, neutral): “, 2021] as well as combining both audio and video inputs [Shi et al., 2022].”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “Some typical approaches in this category include Multi-view autoencoders and variations [154], [155], Multi-modal deep Boltzmann machines [156], Canonical correlation analysis (CCA) [157] and its nonlinear extensions [158], [159], [160], [161], [162], [163], [164], [165], [166], Multi-view contrasti”
Multimodal Learning With Transformers: A Survey (2022, methodology): “audio frame (mel-spectrogram) CNN embedding [173], AVSA [145] audio waveform segment linear projection VATT [174] audio spectrogram patch linear projection MBT [175] speech/audio waveform segment 1D-CNN (TCN) embedding [176], FaceFormer [39] speech frame (mel-spectrogram) linear projection and gated”
ReVISE: Self-Supervised Speech Resynthesis with Visual Input for Universal and Generalized Speech Enhancement (2022, methodology): “1, because the audio quality generated by SVTS is mediocre (2.12 and 2.23 MOS), 3 these samples are not very intelligible by machines, reﬂected by the high WER (81.5% and 67.4% when trained on LRS3 and LRS3+VoxCeleb2).”