Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoders

By Andy T. Liu, Shu-Wen Yang, Po-Han Chi, Po-Chun Hsu, Hung-yi Lee. 2019

403 citations recorded by the source corpus.

We present Mockingjay as a new speech representation learning approach, where bidirectional Transformer encoders are pre-trained on a large amount of unlabeled speech. Previous speech representation methods learn through conditioning on past frames and predicting information about future frames. Whereas Mockingjay is designed to predict the current frame through jointly conditioning on both past and future contexts. The Mockingjay representation improves performance for a wide range of downstream tasks, including phoneme classification, speaker recognition, and sentiment classification on spoken content, while outperforming other approaches. Mockingjay is empirically powerful and can be fine-tuned with downstream models, with only 2 epochs we further improve performance dramatically. In a low resource setting with only 0.1% of labeled data, we outperform the result of Mel-features that uses all 100% labeled data.

Subfields: Artificial Intelligence, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

In Signal Processing

Citations in the corpus

On Generative Spoken Language Modeling from Raw Audio (2021, neutral): “Examples of information include spectrograms (Chung et al., 2019; Wang et al., 2020; Chi et al., 2021; Liu et al., 2020; Chung and Glass, 2020; Liu et al., 2020; Ling et al., 2020; Ling and Liu, 2020), cluster indices (Baevski et al., 2019; Hsu et al., 2021), derived signal processing features…”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Significant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are fine-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
SUPERB: Speech processing Universal PERformance Benchmark (2021, background): “Instances of generative modeling investigated here include APC [7], VQ-APC [32], Mockingjay [8], TERA [9], NPC [33] and DeCoAR 2.0 [10].”
TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech (2020, methodology): “Besides, speech representations also help transfer learning and adaptation across different data distributions [6], [7], [9], [20], [21].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “In speech, instead of prediction, we randomly mask temporal slices of acoustic features and attempt to reconstruct them [21, 22, 23, 24, 25, 16].”
Vector-Quantized Autoregressive Predictive Coding (2020, methodology): “eam tasks. Recently, self-supervised learning—a paradigm that treats the input itself or modiﬁcations of the input as learning targets—has obtained promising results for learning such transformations [1, 2, 4, 5, 6, 7, 8, 9, 3, 10]. These methods, mostly inspired by the techniques for pre-training N”
Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependencies (2020, methodology): “NPC with input mask size M in = 5, receptive ﬁeld size R = 27. Single MaskedConv indicates applying Masked ConvBlock at the last layer only. Evaluation of representation. We follow the previous works [1, 6, 9] to deﬁne the “effectiveness” of representations as the accessibility to hidden information”
FRILL: A Non-Semantic Speech Embedding for Mobile Devices (2020, neutral): “Representation learning is a powerful tool for leveraging large collections of unlabeled data to learn better supervised models when labels are scarce [1, 2, 3].”
The Zero Resource Speech Challenge 2021: Spoken Language Modelling (2021, methodology): “System HL [27] trains representations using Mockingjay, an approach based on bidirectional self-attention [28].”
A Convolutional Deep Markov Model for Unsupervised Speech Representation Learning (2020, background): “Although, generative models are theoretically elegant and informed by theories of cognition, most recent success in speech representation learning has come from self-supervised learning algorithms such as Wav2Vec [8], Problem Agnostic Speech Encoding (PASE) [9], Autoregressive Predictive Coding (APC”
A Comprehensive Survey on Applications of Transformers for Deep Learning Tasks (2023, background): “Mockingjay (Liu et al., 2020) Speech Classification & recognition 2020 Encoder Yes LibriSpeech LibriSpeech test clean, MOSEI”
Masked Autoencoders that Listen (2022, background): “Mockingjay [42] proposed a masked acoustic model pretext task to reconstruct frame-level Mel-features of masked time frames.”
Semi-Supervised Spoken Language Understanding via Self-Supervised Speech and Language Model Pretraining (2020, background): “Notably, self-supervised speech representation learning [21, 22, 23, 24, 25] provides a clean and general learning mechanism for downstream speech tasks, yet the semantic transferrability of these representations are unclear.”