TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech

By Andy T. Liu, Shang-Wen Li, Hung-yi Lee. 2020

414 citations recorded by the source corpus.

We introduce a self-supervised speech pre-training method called TERA, which stands for Transformer Encoder Representations from Alteration. Recent approaches often learn by using a single auxiliary task like contrastive prediction, autoregressive prediction, or masked reconstruction. Unlike previous methods, we use alteration along three orthogonal axes to pre-train Transformer Encoders on a large amount of unlabeled speech. The model learns through the reconstruction of acoustic frames from their altered counterpart, where we use a stochastic policy to alter along various dimensions: time, frequency, and magnitude. TERA can be used for speech representations extraction or fine-tuning with downstream models. We evaluate TERA on several downstream tasks, including phoneme classification, keyword spotting, speaker recognition, and speech recognition. We present a large-scale comparison of various self-supervised models. TERA achieves strong performance in the comparison by improving upon surface features and outperforming previous models. In our experiments, we study the effect of applying different alteration techniques, pre-training on more data, and pre-training on various features. We analyze different model sizes and find that smaller models are strong representation learners than larger models, while larger models are more effective for downstream fine-tuning than smaller models. Furthermore, we show the proposed method is transferable to downstream datasets not used in pre-training.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, methodology): “Self-supervised learning methods are categorized into discriminative methods [1, 2, 3, 4, 5, 6, 7], generative methods [8, 9, 10, 11, 12, 13], and multi-task learning methods [14].”
A3T: Alignment-Aware Acoustic and Text Pretraining for Speech Synthesis and Editing (2022, positive): “Introduction Recently, speech representation learning has attracted much attention in the speech community due to its strong performance to many speech-related downstream tasks, such as speech recognition, speech classification, and speech translation (Baevski et al., 2020; Chen et al., 2020; Liu et”
w2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Training (2021, background): “Signiﬁcant recent research effort has been made to develop proxy tasks that allow models to perform well when the models are ﬁne-tuned on ASR tasks [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20].”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, background): “In speech, instead of prediction, we randomly mask temporal slices of acoustic features and attempt to reconstruct them [21, 22, 23, 24, 25, 16].”
data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “2 Another line of work directly reconstructs the input features (Eloff et al., 2019; Liu et al., 2021).”
Fused Acoustic and Text Encoding for Multimodal Bilingual Pretraining and Speech Translation (2021, neutral): “…existing methods can only handle one modality, either text or speech, while joint acoustic and text representation is desired for many end-to-end spoken language processing tasks, such as spoken question answering (Chuang et al., 2019) and end-to-end speech-to-text translation (Liu et al., 2020b).”
SPLAT: Speech-Language Joint Pre-Training for Spoken Language Understanding (2021, positive): “To boost its capacity for contextual understanding, we borrow the idea of masked language modeling (MLM) (Devlin et al., 2019; Liu et al., 2020c; Wang et al., 2020; Liu et al., 2020b).”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “Whereas most studies use masking along the temporal dimension of the input, speech can also be masked along the frequency dimension when spectral input features are used [103], [106].”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, methodology): “Other studies [12, 13, 14, 15, 16, 17] were motivated by NLP and applied similar methods on speech tasks.”
SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilities (2022, positive): “…their capabilities across tasks like phoneme recognition (van den Oord et al., 2018; Chung et al., 2019), automatic speech recognition (ASR) (Liu et al., 2020b; Schneider et al., 2019; Ling and Liu, 2020; Ravanelli et al., 2020; Hsu et al., 2021; Chang et al., 2021), speaker veriﬁcation…”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “…the signal, using standard prediction losses such as L2 distance or cross-entropy (Chung et al. 2019; Chung, Tang, and Glass 2020; Liu et al. 2020; Liu, Li, and Lee 2021; Ling and Liu 2020; Hsu et al. 2021a; Chen et al. 2021), using a contrastive objective (v. d. Oord, Li, and Vinyals 2018;…”
Scaling Effect of Self-Supervised Speech Models (2021, background): “[11] A. T. Liu, S. Li, and H. Lee, “TERA: self-supervised learning of transformer encoder representation for speech,” CoRR, vol. abs/2007.06028, 2020.”
A Brief Overview of Unsupervised Neural Speech Representation Learning (2022, background): “In the literature, these are often jointly referred to as reconstruction tasks (REC, table 1) (Liu, Li, and Lee 2021; Wang et al. 2021), although this is somewhat misleading in the case of prediction.”