Learning Speaker Representations with Mutual Information

By Mirco Ravanelli, Yoshua Bengio. 2018

96 citations recorded by the source corpus.

Learning good representations is of crucial importance in deep learning. Mutual Information (MI) or similar measures of statistical dependence are promising tools for learning these representations in an unsupervised way. Even though the mutual information between two random variables is hard to measure directly in high dimensional spaces, some recent studies have shown that an implicit optimization of MI can be achieved with an encoder-discriminator architecture similar to that of Generative Adversarial Networks (GANs). In this work, we learn representations that capture speaker identities by maximizing the mutual information between the encoded representations of chunks of speech randomly sampled from the same sentence. The proposed encoder relies on the SincNet architecture and transforms raw speech waveform into a compact feature vector. The discriminator is fed by either positive samples (of the joint distribution of encoded chunks) or negative samples (from the product of the marginals) and is trained to separate them. We report experiments showing that this approach effectively learns useful speaker representations, leading to promising results on speaker identification and verification tasks. Our experiments consider both unsupervised and semi-supervised settings and compare the performance achieved with different objective functions.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Multi-Task Self-Supervised Learning for Robust Speech Recognition (2020, neutral): “Recently, a similar paradigm has been used to infer audio representations [11, 12], including learning speech representations with mutual information [13, 14].”
Improving Transformer-based Speech Recognition Using Unsupervised Pre-training (2019, background): “There are multiple concurrent approaches that generalize this approach and applied it in learning speaker representation [8], extracting speech representation [5, 9] and performing speech emotion recognition [6].”
wav2vec: Unsupervised Pre-training for Speech Recognition (2019, neutral): “In speech processing, pre-training has focused on emotion recogniton (Lian et al., 2018), speaker identiﬁcation (Ravanelli & Bengio, 2018), phoneme discrimination (Synnaeve & Dupoux, 2016a; van den Oord et al., 2018) as well as transferring ASR representations from one language to another (Kunze et…”
Learning Problem-agnostic Speech Representations from Multiple Self-supervised Tasks (2019, methodology): “Some attempts have also been done to extend self-supervised learning to different modalities [12,13] or to audio representations only [14–17].”
Pretraining Techniques for Sequence-to-Sequence Voice Conversion (2020, background): “Efforts have also been dedicated to speech representation learning for tasks including automatic speech recognition (ASR) [27] and speaker identification/verification [28].”
End-to-End anti-spoofing with RawNet2 (2020, background): “One solution to these issues is SincNet [13, 14].”
Self-Supervised Speech Representation Learning: A Review (2022, background): “ConvDMM [235] LS 50 [234]/360/ 960 hrs PC/PR WSJ si284 WSJ eval92 X 5(12)/50(12)/100(12) mins, 4(12)/8(12)/40(12) hrs”
Improved RawNet with Filter-wise Rescaling for Text-independent Speaker Verification using Raw Waveforms (2020, methodology): “We also investigated replacing the RawNet’s first convolutional layer with a sinc-convolution (sinc-conv) layer, which was first proposed to process raw waveforms by performing time-domain convolutions [4, 12].”
BEATs: Audio Pre-Training with Acoustic Tokenizers (2022, background): “LIM [Ravanelli and Bengio, 2018], COLA [Saeed et al., 2021] and Fonseca et al. [2021] adopt the contrastive learning framework for audio pre-training, where the positive samples are the augmented clips from the same audio, and the negative ones are sampled from the different audios.”
OPT: Omni-Perception Pre-Trainer for Cross-Modal Understanding and Generation (2021, background): “In audio and speech processing, pre-training has focused on emotion recognition [24], speaker identification [32], phoneme discrimination [39, 27], transferring ASR representations from one language to another [17], unsupervised representations learning for speech [35], audio representation learning”
Speech Recognition (2018, unknown): “In speech processing, pre-training has focused on emotion recogniton (Lian et al., 2018), speaker identification (Ravanelli & Bengio, 2018), phoneme discrimination (Synnaeve & Dupoux, 2016a; van den Oord et al., 2018) as well as transferring ASR representations from one language to another (Kunze…”
Whitening for Self-Supervised Representation Learning (2020, background): “re a large number of samples. Moreover, our loss formulation is also simpler since it does not require a proper setting of the ˝ hyperparameter in Eq. 2 or min Eq. 1. Finally, while many recent works [33,21,23,1,36] draw a relation between the contrastive loss and an estimate of the mutual informati”
A Further Study of Unsupervised Pretraining for Transformer Based Speech Recognition (2020, methodology): “Multiple concurrent approaches generalized this approach [2, 3] and applied it in learning speaker representation [4], extracting speech representations [5, 6], conducting various speech-related tasks such as speech recognition [5, 7, 8] and speech emotion recognition [9].”