Learning Robust and Multilingual Speech Representations

By Kazuya Kawakami, Luyu Wang, Chris Dyer, Phil Blunsom, Aäron van den Oord. 2020

103 citations recorded by the source corpus.

Unsupervised speech representation learning has shown remarkable success at finding representations that correlate with phonetic structures and improve downstream speech recognition performance. However, most research has been focused on evaluating the representations in terms of their ability to improve the performance of speech recognition systems on read English (e.g. Wall Street Journal and LibriSpeech). This evaluation methodology overlooks two important desiderata that speech representations should have: robustness to domain shifts and transferability to other languages. In this paper we learn representations from up to 8000 hours of diverse and noisy speech data and evaluate the representations by looking at their robustness to domain shifts and their ability to improve recognition performance in many languages. We find that our representations confer significant robustness advantages to the resulting recognition systems: we see significant improvements in out-of-domain transfer relative to baseline feature sets and the features likewise provide improvements in 25 phonetically diverse languages.

Subfields: Natural Language Processing, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Natural Language Processing

In Representation & Self-Supervised Learning

Citations in the corpus

Contrastive Multiview Coding (2019, background): “The idea of contrastive learning has started to spread over many other tasks in various other domains [73,86,60,76,47,37,74,81].”
wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations (2020, neutral): “The latent representations are fed to a Transformer network to build contextualized representations and the model is trained via a contrastive task where the true latent is to be distinguished from distractors [54, 49, 48, 28] (§ 2).”
mSLAM: Massively multilingual joint pre-training for speech and text (2022, background): “Similarly, in speech understanding, multilingual pre-trained models (Kawakami et al., 2020; Conneau et al., 2020; Babu et al., 2021) based on self-supervised losses (Oord et al.”
XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale (2021, background): “For speech processing, self-supervised approaches such as wav2vec 2.0 [7, 12] have also been extended to the multilingual setting [13, 14].”
Unsupervised Speech Recognition (2021, background): “There has also been work on robustness of self-supervised learning to domain shift (Hsu et al., 2021a), multilingual self-supervised speech representation learning (Kawakami et al., 2020; Conneau et al., 2020) as well as work on combining modalities such as speech and vision (Harwath et al., 2020).”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, background): “Furthermore, the pre-trained feature extractor in [17] is fixed during supervised fine-tuning, making it more similar to featurebased approaches.”
Unsupervised Pretraining Transfers Well Across Languages (2020, background): “Further work needs to be done to establish how these pretrained features can be best used in the low resource setting (see [30]), and with other ASR tasks [29].”
Unsupervised Cross-lingual Representation Learning for Speech Recognition (2020, background): “Different to recent work on unsupervised cross-lingual pretraining, we fine-tune the Transformer part of the model instead of freezing all pretrained representations [27] or feeding them to a separate downstream model [28].”
Data Augmenting Contrastive Learning of Speech Representations in the Time Domain (2020, neutral): “The encoder architecture is unchanged (5 convolutional layers with kernel sizes [10,8,4,4,4], strides [5,4,2,2,2] and hidden dimension 256).”
The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling (2020, methodology): “…of raw audio, giving rise to very effective methods of pretraining for downstream ASR tasks, particularly in the low resource scenario (Schneider et al., 2019; Baevski et al., 2019; Chung and Glass, 2019; Baevski et al., 2020b; Rivière et al., 2020; Kawakami et al., 2020; Wang et al., 2020).”
Contrastive Learning of General-Purpose Audio Representations (2020, background): “This approach learns a representation from pairs of similar inputs from unlabeled data, exploiting e.g. temporal consistency [5, 4, 6] or data augmentation [7] and trains a model to recognize similar elements among negative distractors.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “Some work has extended APC and CPC inspired models with separate encoders operating in opposite directions (Ling et al. 2020; Kawakami et al. 2020; Borgholt et al. 2021b), but these models are still restricted to process left and right context separately.”
Multimodal Self-Supervised Learning of General Audio Representations (2021, unknown): “It can leverage large amounts of unlabelled data and produce competitive performance on vision, language, and speech tasks [5, 6, 7, 8, 9, 10, 11, 12].”
Multi-Format Contrastive Learning of Audio Representations (2021, methodology): “Different from the models that drive the current progress on learning unsupervised speech representations [9, 10, 11, 12, 13, 14], our method does not rely on the context prediction network, and directly contrast two augmented views (which resembles image models in [5, 6]).”
Automatic speaker verification spoofing and deepfake detection using wav2vec 2.0 and data augmentation (2022, unknown): “Examples include contrastive predictive coding (CPC) [16, 29], auto-regressive predictive coding [30], wav2vec [31], HuBERT [32,33], wav2vec 2.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “and conventionally used for evaluation, while their audio streams are also aggregated to build diversified and large-scale corpora for unsupervised pre-training [109], [123], [218].”