What Does the Speaker Embedding Encode?

By Shuai Wang, Yanmin Qian, Kai Yu. 2017

61 citations recorded by the source corpus.

Developing a good speaker embedding has received tremendous interest in the speech community, with representations such as i-vector and d-vector demonstrating remarkable performance across various tasks. Despite their widespread adoption, a fundamental question remains largely unexplored: what properties are actually encoded in these embeddings? To address this gap, we conduct a comprehensive analysis of three prominent speaker embedding methods: i-vector, d-vector, and RNN/LSTM-based sequence-vector (s-vector). Through carefully designed classification tasks, we systematically investigate their encoding capabilities across multiple dimensions, including speaker identity, gender, speaking rate, text content, word order, and channel information. Our analysis reveals distinct strengths and limitations of each embedding type: i-vector excels at speaker discrimination but encodes limited sequential information; s-vector captures text content and word order effectively but struggles with speaker identity; d-vector shows balanced performance but loses sequential information through averaging. Based on these insights, we propose a novel multi-task learning framework that integrates i-vector and s-vector, resulting in a new speaker embedding (i-s-vector) that combines their complementary advantages. Experimental results on RSR2015 demonstrate that the proposed i-s-vector achieves more than 50% EER reduction compared to the i-vector baseline on content mismatch trials, validating the effectiveness of our approach.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

In Speech & Audio

In Signal Processing

Citations in the corpus

Analysis Methods in Neural Language Processing: A Survey (2018, background): “Much recent work has focused on visualizing activations on specific examples in modern neural networks for language (Karpathy et al., 2015; Kádár et al., 2017; Qian et al., 2016a; Liu et al., 2018) and speech (Wu and King, 2016; Nagamine et al., 2015; Wang et al., 2017b).”
Analyzing Hidden Representations in End-to-End Automatic Speech Recognition Systems (2017, neutral): “Recent work analyzed different speaker representations [17].”
Analyzing Learned Representations of a Deep ASR Performance Prediction Model (2018, background): “Also relevant is the work of (Wang et al., 2017) who proposed an in-depth investigation on three kinds of speaker embeddings learned for a speaker recognition task, i.e. i-vector, d-vector and RNN/LSTM based sequence-vector (s-vector).”
On internal language representations in deep learning: an analysis of machine translation and speech recognition (2018, methodology): “A more common approach is to predict certain linguistic properties from activations of the neural network [2, 106, 117, 123, 188, 252, 278, 279, 306, 333].”