To Reverse the Gradient or Not: an Empirical Comparison of Adversarial and Multi-task Learning in Speech Recognition

By Yossi Adi, Neil Zeghidour, Ronan Collobert, Nicolas Usunier, Vitaliy Liptchinsky, Gabriel Synnaeve. 2018

44 citations recorded by the source corpus.

Transcribed datasets typically contain speaker identity for each instance in the data. We investigate two ways to incorporate this information during training: Multi-Task Learning and Adversarial Learning. In multi-task learning, the goal is speaker prediction; we expect a performance improvement with this joint training if the two tasks of speech recognition and speaker recognition share a common set of underlying features. In contrast, adversarial learning is a means to learn representations invariant to the speaker. We then expect better performance if this learnt invariance helps generalizing to new speakers. While the two approaches seem natural in the context of speech recognition, they are incompatible because they correspond to opposite gradients back-propagated to the model. In order to better understand the effect of these approaches in terms of error rates, we compare both strategies in controlled settings. Moreover, we explore the use of additional un-transcribed data in a semi-supervised, adversarial learning manner to improve error rates. Our results show that deep models trained on big datasets already develop invariant representations to speakers without any auxiliary loss. When considering adversarial learning and multi-task learning, the impact on the acoustic model seems minor. However, models trained in a semi-supervised manner can improve error-rates.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

1. Wei-Ning Hsu
2. Adam Polyak
3. Evgeny Kharitonov
4. Tu Nguyen
5. Jade Copet
6. Yossi Adi
7. Ann Lee
8. Lior Wolf

Citations in the corpus

• Privacy-Preserving Adversarial Representation Learning in ASR: Reality or Illusion? (2019, neutral): “This question was studied in [30], in which the authors conclude that both approaches only provide minor improvements in terms of ASR performance.”
• textless-lib: a Library for Textless Spoken Language Processing (2022, methodology): “Hence, as the first example, we include a probing experiment similar to the one proposed in (van Niekerk et al., 2021; Adi et al., 2019).”
• TTS Skins: Speaker Conversion via ASR (2019, methodology): “ncoder is taken to be a subnetwork of a pre-trained ASR network, specifically the public implementation [21] of Wav2Letter [22]. Since an ASR network is used, by design, it is invariant to the speaker [23], and the method does not require any disentanglement terms or other domain (speaker) confusion ”
• Any-to-Many Voice Conversion With Location-Relative Sequence-to-Sequence Modeling (2020, background): “The bottle-neck features computed from speech signals are regarded as spoken content representations and are presumed to be speaker-invariant [37].”
• Unsupervised Cross-Domain Singing Voice Conversion (2020, methodology): “Since an ASR network is speaker-agnostic by design [37], our method does not require any disentanglement terms or domain-specific (speaker) confusion terms.”