DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization

By Shaoshi Ling, Yuzong Liu. 2020

117 citations recorded by the source corpus.

Recent success in speech representation learning enables a new way to leverage unlabeled data to train speech recognition model. In speech representation learning, a large amount of unlabeled data is used in a self-supervised manner to learn a feature representation. Then a smaller amount of labeled data is used to train a downstream ASR system using the new feature representations. Based on our previous work DeCoAR and inspirations from other speech representation learning, we propose DeCoAR 2.0, a Deep Contextualized Acoustic Representation with vector quantization. We introduce several modifications over the DeCoAR: first, we use Transformers in encoding module instead of LSTMs; second, we introduce a vector quantization layer between encoder and reconstruction modules; third, we propose an objective that combines the reconstructive loss with vector quantization diversity loss to train speech representations. Our experiments show consistent improvements over other speech representations in different data-sparse scenarios. Without fine-tuning, a light-weight ASR model trained on 10 hours of LibriSpeech labeled data with DeCoAR 2.0 features outperforms the model trained on the full 960-hour dataset with filterbank features.

Subfields: Artificial Intelligence, Natural Language Processing, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Natural Language Processing

In Signal Processing

Citations in the corpus

Scaling Laws for Acoustic Models (2021, methodology): “The two models evaluated in this work share the same basic auto-predictive coding (APC) design, shown in Figure 1.”
HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units (2021, background): “creasing interests recently, where a model is tasked to predict the content of the unseen regions [4], [44]–[50] or to contrast the target unseen frame with randomly sampled ones [1]– [3], [6].”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, background): “Recent works propose to predict future frames from the history with an autoregressive model [24]–[27], or recover the masked frames from the corrupted speech with a non-autoregressive model [28]–[32].”
Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, background): “…is increasing interest in exploring speech pre-training using large quantities of unlabeled speech data (Chung et al., 2019; Schneider et al., 2019; Kharitonov et al., 2021; Baevski et al., 2020; Hsu et al., 2021c; Liu et al., 2020; Ling and Liu, 2020; Tjandra et al., 2020; Hsu et al., 2021b,a).”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Significant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are fine-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
SUPERB: Speech processing Universal PERformance Benchmark (2021, background): “…have investigated these SSL models’ capabilities on tasks including phoneme classiﬁcation [11, 7], speaker identiﬁcation [7, 8], speaker ver-iﬁcation [17], emotion recognition [15], ASR [9, 12, 10, 16], speech translation [7], spoken language understanding [18], voice conversion [19] and TTS [20].”
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021, background): “…et al., 2020a), contrastive predictive coding (Oord et al., 2018; Rivière et al., 2020), autoregressive predictive coding (Chung et al., 2019a, 2020; Chung and Glass, 2020b), and DeCoAR (Ling et al., 2020; Ling and Liu, 2020), to capture contextual representations from unlabeled speech data.”
Self-Supervised Speech Representation Learning: A Review (2022, background): “0 [92] LS 960 hrs ASR LS 100 hrs LS test-clean, LS test-other 1 (12)/10(12)/100 hrs”
Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Prediction (2022, unknown): “0 (Ling & Liu, 2020), HuBERT (Hsu et al., 2021c;a)) are mostly built entirely on audio.”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “…prediction losses such as L2 distance or cross-entropy (Chung et al. 2019; Chung, Tang, and Glass 2020; Liu et al. 2020; Liu, Li, and Lee 2021; Ling and Liu 2020; Hsu et al. 2021a; Chen et al. 2021), using a contrastive objective (v. d. Oord, Li, and Vinyals 2018; Schneider et al. 2019;…”
Generative Pre-training for Speech with Flow Matching (2023, background): “Another branch of works (Ling et al., 2020; Ling & Liu, 2020) studied masked audio modeling (MAM) instead of future prediction.”