Analyzing Speaker Information in Self-Supervised Models to Improve Zero-Resource Speech Processing
By Benjamin van Niekerk, Leanne Nortje, Matthew Baas, Herman Kamper. 2021
37 citations recorded by the source corpus.
Contrastive predictive coding (CPC) aims to learn representations of speech by distinguishing future observations from a set of negative examples. Previous work has shown that linear classifiers trained on CPC features can accurately predict speaker and phone labels. However, it is unclear how the features actually capture speaker and phonetic information, and whether it is possible to normalize out the irrelevant details (depending on the downstream task). In this paper, we first show that the per-utterance mean of CPC features captures speaker information to a large extent. Concretely, we find that comparing means performs well on a speaker verification task. Next, probing experiments show that standardizing the features effectively removes speaker information. Based on this observation, we propose a speaker normalization step to improve acoustic unit discovery using K-means clustering of CPC features. Finally, we show that a language model trained on the resulting units achieves some of the best results in the ZeroSpeech2021~Challenge.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Wei-Ning Hsu
- Yossi Adi
- Emmanuel Dupoux
- Abdel-rahman Mohamed
- James R. Glass
- David F. Harwath
- Jinyu Li
- Shinji Watanabe
Citations in the corpus
- Self-Supervised Speech Representation Learning: A Review (2022, background): “It is shown that the per-utterance mean of CPC features captures speaker information to a large extent [271].”
- The Zero Resource Speech Challenge 2021: Spoken Language Modelling (2021, methodology): “System BN [25] begins with the baseline CPC representations and applies speaker normalization before re-running kmeans.”
- textless-lib: a Library for Textless Spoken Language Processing (2022, methodology): “Hence, as the first example, we include a probing experiment similar to the one proposed in (van Niekerk et al., 2021; Adi et al., 2019).”
- Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modeling (2022, background): “The non-VG track was introduced earlier, and submissions to this track studied the effect of filtering out speaker information (v. Niekerk et al. 2021), denoising CPC (v. d. Oord, Li, and Vinyals 2018) representations by using methods from information retrieval (Chorowski et al. 2021), and combining…”