The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling

By Tu Nguyen, Maureen de Seyssel, Patricia Roz'e, Morgane Rivière, Evgeny Kharitonov, Alexei Baevski, Ewan Dunbar, Emmanuel Dupoux. 2020

135 citations recorded by the source corpus.

We introduce a new unsupervised task, spoken language modeling: the learning of linguistic representations from raw audio signals without any labels, along with the Zero Resource Speech Benchmark 2021: a suite of 4 black-box, zero-shot metrics probing for the quality of the learned models at 4 linguistic levels: phonetics, lexicon, syntax and semantics. We present the results and analyses of a composite baseline made of the concatenation of three unsupervised systems: self-supervised contrastive representation learning (CPC), clustering (k-means) and language modeling (LSTM or BERT). The language models learn on the basis of the pseudo-text derived from clustering the learned representations. This simple pipeline shows better than chance performance on all four metrics, demonstrating the feasibility of spoken language modeling from raw speech. It also yields worse performance compared to text-based 'topline' systems trained on the same data, delineating the space to be explored by more sophisticated end-to-end models.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

SUPERB: Speech processing Universal PERformance Benchmark (2021, methodology): “There are existing benchmarks proposed to evaluate representations extracted from SSL pretrained models [21, 22].”
Are Discrete Units Necessary for Spoken Language Modeling? (2022, methodology): “ation metrics of the Zero Resource Challenge 2021 (Nguyen et al., 2020; Dunbar et al., 2021) on different linguistic levels.”
EXPRESSO: A Benchmark and Analysis of Discrete Expressive Speech Resynthesis (2023, unknown): “…like /bit/ versus /bet/ are more distant to one another than two instances of the same triphone [15]) both on the 1-hot representations (as in [16]), and using the dense embedding corresponding to the centroid of the units, and PNMI (phone-normalized mutual information between units z and…”
Self-Supervised Speech Representation Learning: A Review (2022, background): “The Zero Resource Speech Challenge 2021 [326], the latest challenge, expanded the scope to include language modeling tasks.”
Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing (2021, methodology): “We incorporate speaker normalization into a baseline acoustic unit discovery system built on K-means clustering applied to CPC features [17].”
The Zero Resource Speech Challenge 2021: Spoken Language Modelling (2021, methodology): “The language models are based on a smaller BERT recipe (28M parameters) we provided [4].”
VoxtLM: Unified Decoder-Only Models for Consolidating Speech Recognition, Synthesis and Speech, Text Continuation Tasks (2023, methodology): “• For speech and text generation, we use sWUGGY and sBLIMP using dev dataset as proposed in [30] and perplexity (PPL).”
Layer-Wise Analysis of a Self-Supervised Speech Representation Model (2021, background): “[7] T. A. Nguyen et al., “The Zero Resource Speech Benchmark 2021: Metrics and baselines for unsupervised spoken language modeling,” arXiv:2011.11588, 2020.”
Textually Pretrained Speech Language Models (2023, methodology): “We use the sWUGGY and sBLIMP [Nguyen et al., 2020] metrics to evaluate lexical and syntactic modeling of the SpeechLMs.”
Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, methodology): “To better measure the progress within this research area, two benchmarks, the ZeroSpeech 2021 challenge (Nguyen et al. 2020; Alishahi et al. 2021) and the SUPERB benchmark (Yang et al. 2021), were recently proposed.”
textless-lib: a Library for Textless Spoken Language Processing (2022, background): “There are plenty of ways to evaluate the overall pipeline (Lakhotia et al., 2021; Dunbar et al., 2019, 2020; Nguyen et al., 2020) and different ways to model the “pseudo-text” units (Shi et al., 2021; Kharitonov et al., 2021a; Polyak et al., 2021; Kreuk et al., 2021; Lee et al., 2021a), hence…”