Layer-Wise Analysis of a Self-Supervised Speech Representation Model
By Ankita Pasad, Ju-Chieh Chou, Karen Livescu. 2021
472 citations recorded by the source corpus.
Recently proposed self-supervised learning approaches have been successful for pre-training speech representation models. The utility of these learned representations has been observed empirically, but not much has been studied about the type or extent of information encoded in the pre-trained representations themselves. Developing such insights can help understand the capabilities and limits of these models and enable the research community to more efficiently develop their usage for downstream applications. In this work, we begin to fill this gap by examining one recent and successful pre-trained model (wav2vec 2.0), via its intermediate representation vectors, using a suite of analysis tools. We use the metrics of canonical correlation, mutual information, and performance on simple downstream tasks with non-parametric probes, in order to (i) query for acoustic and linguistic information content, (ii) characterize the evolution of information across model layers, and (iii) understand how fine-tuning the model for automatic speech recognition (ASR) affects these observations. Our findings motivate modifying the fine-tuning protocol for ASR, which produces improved word error rates in a low-resource setting.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Wei-Ning Hsu
- Michael Auli
- Alexei Baevski
- Abdel-rahman Mohamed
- James R. Glass
- Jinyu Li
- Shinji Watanabe
- Hung-yi Lee
Citations in the corpus
- StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Models (2023, background): “Speech language models (SLMs) encode valuable information ranging from acoustic to semantic aspects [55], and SLM representations are shown to mimic human perception for evaluating synthesized speech quality [45].”
- AudioLM: A Language Modeling Approach to Audio Generation (2022, background): “, phonemes and word identity when trained on speech [34]).”
- data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Language (2022, background): “This idea was partly inspired by the fact that the top layers of wav2vec 2.0 do not perform as well for downstream tasks as layers in the middle of the network (Baevski et al., 2021; Pasad et al., 2021).”
- Self-Supervised Speech Representation Learning: A Review (2022, background): “[68] further show that fine-tuning the pre-trained model with a character-level CTC training criterion changes the behavior of the last layers (especially the final two layers), breaking”