Exploring the Encoding Layer and Loss Function in End-to-End Speaker and Language Recognition System

By Weicheng Cai, Jinkun Chen, Ming Li. 2018

356 citations recorded by the source corpus.

In this paper, we explore the encoding/pooling layer and loss function in the end-to-end speaker and language recognition system. First, a unified and interpretable end-to-end system for both speaker and language recognition is developed. It accepts variable-length input and produces an utterance level result. In the end-to-end system, the encoding layer plays a role in aggregating the variable-length input sequence into an utterance level representation. Besides the basic temporal average pooling, we introduce a self-attentive pooling layer and a learnable dictionary encoding layer to get the utterance level representation. In terms of loss function for open-set speaker verification, to get more discriminative speaker embedding, center loss and angular softmax loss is introduced in the end-to-end system. Experimental results on Voxceleb and NIST LRE 07 datasets show that the performance of end-to-end learning system could be significantly improved by the proposed encoding layer and loss function.

Subfields: Artificial Intelligence, Model Architectures, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Speech & Audio

Citations in the corpus

VOXLINGUA107: A Dataset for Spoken Language Recognition (2020, methodology): “For frame-level feature extraction, we use the Resnet34 [15, 16] architecture where the basic convolutional blocks with residual connections are replaced with squeeze-and-attention modules [17, 18].”
Neural Codec Language Models are Zero-Shot Text to Speech Synthesizers (2023, positive): “To improve the quality of unseen speakers, advanced speaker embedding models [Cai et al., 2018] can be employed, but it is still undesirable according to Tan et al.”
YourTTS: Towards Zero-Shot Multi-Speaker TTS and Zero-Shot Voice Conversion for everyone (2021, methodology): “Similarly, Tacotron 2 was used with a different speaker embeddings methods [2], with LDE embeddings [9] to improve similarity and naturalness of speech for unseen speakers [10].”
Zero-Shot Multi-Speaker Text-To-Speech with State-Of-The-Art Neural Speaker Embeddings (2019, methodology): “For the LDEs, our ResNet34 is the same as [16, 19, 21], and we set the number of dictionary clusters C = 32 and minibatch size to 128.”
SC-GlowTTS: an Efficient Zero-Shot Multi-Speaker Text-To-Speech Model (2021, background): “The authors showed that LDE [8] embeddings improved the similarity and synthesized a more natural speech for novel speakers when compared to X-vector [9] embeddings.”
Can Speaker Augmentation Improve Multi-Speaker End-to-End TTS? (2020, methodology): “Therefore, similar to our speaker encoder network, we reused the Learnable Dictionary Encoding (LDE) [28] based network architecture for our dialect encoder network.”
AutoSpeech: Neural Architecture Search for Speaker Recognition (2020, unknown): “Additionally, advanced training objectives [2, 8, 29] and temporal aggregation strategies [7, 3, 9, 5] are also proposed to improve speaker recognition performance.”
Exploring wav2vec 2.0 on speaker verification and language identification (2020, unknown): “EER I-vectors + PLDA [24] 8.8 CNN + Embedding [24] 7.8 LDE-ASoftmax [27] 4.41 attentive statistics [28] 3.85 no pre-training 24.28 M-sv”
From Speaker Verification to Multispeaker Speech Synthesis, Deep Transfer with Feedback Constraint (2020, background): “In the speaker verification field, deep speaker feature learning systems proposed in recent years have achieved comparable performance or even surpassed the classical i-vector systems [16, 17, 18].”
Utterance-level Aggregation for Speaker Recognition in the Wild (2019, methodology): “SAP: Self-attentive Pooling Layer [15], † Cleaned up versions of the test lists have been released publically.”
Learning Disentangled Phone and Speaker Representations in a Semi-Supervised VQ-VAE Paradigm (2020, background): “Speaker VQ codes can also be used conveniently for voice conversion, as this would not require external representations (such as x-vector or LDE).”
Clova Baseline System for the VoxCeleb Speaker Recognition Challenge 2020 (2020, methodology): “Q / SAP AM-Softmax 7 3 2.20 0.139 2.10 0.137 3.67 0.213 2.31 0.144 5.43 0.295 Q / SAP AAM-Softmax 7 3 2.13 0.138 2.12 0.140 3.52 0.211 2.33 0.145 5.19 0.290”
In defence of metric learning for speaker recognition (2020, methodology): “Self-attentive pooling (SAP) [34] is used to aggregate frame-level features into utterance-level representation while paying attention to the frames that are more informative for utterance-level speaker recognition.”
Speaker recognition based on deep learning: An overview (2020, background): “proposed a learnable dictionary encoding (LDE) pooling layer which models the distribution of the frame-level features H by a dictionary [122].”
Improved Prosody from Learned F0 Codebook Representations for VQ-VAE Speech Waveform Reconstruction (2020, methodology): “The speaker condition is represented using a neural embedding vector obtained from a speaker encoder, which was pre-trained using the Learnable Dictionary Encoding (LDE) and angular softmax [15].”
Delving into VoxCeleb: environment invariant speaker recognition (2019, background): “Unlike the TAP layer that equally pools the features over time, [34] introduces a self-attentive pooling layer to pay attention to the frames that are more informative for utterance-level speaker recognition.”
Voxceleb: Large-scale speaker verification in the wild (2020, methodology): “Typically, such systems are trained end-to-end for classi ﬁ cation with a softmax loss (Okabe et al., 2018) or one of its variants, such as the angular softmax (Cai et al., 2018c).”
Deep Speaker Embedding Extraction with Channel-Wise Feature Responses and Additive Supervision Softmax Loss Function (2019, background): “80 Cai et al.[22] ResNet-34 A-Softmax PLDA VoxCeleb1 Pytorch 4.”