SUPERB: Speech processing Universal PERformance Benchmark

By Shu-Wen Yang, Po-Han Chi, Yung-Sung Chuang, Cheng-I Lai, Kushal Lakhotia, Yist Y. Lin, Andy T. Liu, Jiatong Shi, Xuankai Chang, Guan-Ting Lin and 10 more. 2021

1,225 citations recorded by the source corpus.

Self-supervised learning (SSL) has proven vital for advancing research in natural language processing (NLP) and computer vision (CV). The paradigm pretrains a shared model on large volumes of unlabeled data and achieves state-of-the-art (SOTA) for various tasks with minimal adaptation. However, the speech processing community lacks a similar setup to systematically explore the paradigm. To bridge this gap, we introduce Speech processing Universal PERformance Benchmark (SUPERB). SUPERB is a leaderboard to benchmark the performance of a shared model across a wide range of speech processing tasks with minimal architecture changes and labeled data. Among multiple usages of the shared model, we especially focus on extracting the representation learned from SSL due to its preferable re-usability. We present a simple framework to solve SUPERB tasks by learning task-specialized lightweight prediction heads on top of the frozen shared model. Our results demonstrate that the framework is promising as SSL representations show competitive generalizability and accessibility across SUPERB tasks. We release SUPERB as a challenge with a leaderboard and a benchmark toolkit to fuel the research in representation learning and general speech processing.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Speech & Audio

Citations in the corpus

Unispeech-Sat: Universal Speech Representation Learning With Speaker Aware Pre-Training (2021, neutral): “The improved performance on different speech tasks in SUPERB benchmark [15] also veriﬁes the effectiveness of pre-training.”
WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing (2021, methodology): “We compare our WavLM with several SSL models which are evaluated by [8].”
SpeechT5: Unified-Modal Encoder-Decoder Pre-Training for Spoken Language Processing (2021, methodology): “We also list the scores reported in SUPERB (Yang et al., 2021) and SpeechNet (Hsu et al.”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, background): “Self-supervised speech encoder pre-training has led to huge performance improvement on a wide range of applications [22] such as ASR [20, 23], S2T [18], speaker identification [24], etc.”
Generative Spoken Dialogue Language Modeling (2022, background): “…Oord et al., 2018; APC: Chung and Glass, 2020; wav2vec 2.0: Baevski et al., 2020; HuBERT: Hsu et al., 2021a; MockingJay: Liu et al., 2020) from raw speech can learn audio representation that can be used for a variety of downstream tasks (Yang et al., 2021), see Borgholt et al. (2022) for a review.”
Text-Free Prosody-Aware Generative Spoken Language Modeling (2021, background): “However, most of the studies evaluate their models on discriminative tasks, such as ASR and those in the SUPERB benchmark (Yang et al., 2021).”
Direct Speech-to-Speech Translation With Discrete Units (2021, methodology): “to use self-supervised representations from HuBERT [7] to generate the target discrete units for our task, since [5, 10, 11] have shown its superior performance across ASR, spoken language modeling and speech synthesis, compared to other unsupervised representations, including VAE-based representati”
GigaST: A 10, 000-hour Pseudo Speech Translation Corpus (2022, methodology): “4 on LibriSpeech [28], but in ST tasks, we recommend fine-tuning the speech encoders together with other components, which can greatly improve ST performance.”
Disentangled Feature Learning for Real-Time Neural Speech Coding (2022, methodology): “For disentanglement, we leverage the voice conversion benchmark, S3PRLVC [17], an extension of the SUPERB [18] toolkit that focuses on the voice conversion (VC) downstream task, to evaluate the discrete content features.”
Generative Pre-training for Speech with Flow Matching (2023, methodology): “Self-supervised representation learning methods are found to be useful in many different applications such as speech recognition (Yang et al., 2021).”
SpeechLM: Enhanced Speech Pre-Training With Unpaired Textual Data (2022, methodology): “For SUPERB evaluation, we follow the setup in [10].”
SSAST: Self-Supervised Audio Spectrogram Transformer (2021, methodology): “For the three benchmarks Speech Commands V1, Vox-Celeb1, and IEMOCAP that the original AST has not been tested on, we use the standard SUPERB (Yang et al. 2021) training and testing framework.”
Zero-Shot Text-to-Speech Synthesis Conditioned Using Self-Supervised Speech Representation Model (2023, methodology): “The key idea is the direct use of an SSL model to obtain speaker embeddings in the same manner as models in [15].”
emotion2vec: Self-Supervised Pre-Training for Speech Emotion Representation (2023, neutral): “We follow the evaluation of SUPERB (Yang et al., 2021), freezing the pre-trained model and training downstream linear layers with the hidden dimensional set to 256.”