Learning Noise-Invariant Representations for Robust Speech Recognition

By Davis Liang, Zhiheng Huang, Zachary Chase Lipton. 2018

58 citations recorded by the source corpus.

Despite rapid advances in speech recognition, current models remain brittle to superficial perturbations to their inputs. Small amounts of noise can destroy the performance of an otherwise state-of-the-art model. To harden models against background noise, practitioners often perform data augmentation, adding artificially-noised examples to the training set, carrying over the original label. In this paper, we hypothesize that a clean example and its superficially perturbed counterparts shouldn’t merely map to the same class — they should map to the same representation. We propose invariant-representation-learning (IRL): At each training iteration, for each training example, we sample a noisy counterpart. We then apply a penalty term to coerce matched representations at each layer (above some chosen layer). Our key results, demonstrated on the LibriSpeech dataset are the following: (i) IRL significantly reduces character error rates (CER) on both ‘clean’ (3.3% vs 6.5%) and ‘other’ (11.0% vs 18.1%) test sets; (ii) on several-of-domain noise settings (different from those seen during training) IRL’s benefits are even more pronounced. Careful ablations confirm that our results are not simply due to shrinking activations at the chosen layers.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Julian Salazar
2. Jinyu Li
3. Toan Q. Nguyen
4. Dong Yu
5. Li Deng
6. Yifan Gong
7. Zhuo Chen
8. Graham Neubig

In Signal Processing

1. Julian Salazar
2. Jinyu Li
3. Toan Q. Nguyen
4. Dong Yu
5. Yifan Gong
6. Li Deng
7. Zhong Meng
8. Zhuo Chen

Citations in the corpus

• Transformers without Tears: Improving the Normalization of Self-Attention (2019, background): “This trend is clearest in the decoder, where g linearly scales up to the output layer, perhaps in tandem with the discriminativeness of the hidden representations (Liang et al., 2018).”
• Optimal Completion Distillation for Sequence Learning (2018, neutral): “0 Invariant Representation Learning (Liang et al., 2018) 3.”
• Recent Advances in End-to-End Automatic Speech Recognition (2021, background): “Noise-invariant feature was learned to improve robustness in [165, 331].”
• Invariant representation learning for robust deep networks (2018, methodology): “Formulations can be very simple: IRL-E [7] supervises the encoded representations of an original and a noised speech sample; explicitly, α = β = γ = K = 1 , with γ (cid:96) = 0 for all but the last encoder layer (cid:96) = e .”
• Self-attention Networks for Connectionist Temporal Classification in Speech Recognition (2019, unknown): “[9] chr. 6.9 7.2 — — CTC (DS1-like) [33,43] chr. — 6.5 — — Enc-Dec (4-4) [44] chr. 6.5 — 18.1 — Enc-Dec (6-1) [45] chr. 4.5 — 11.6 — CTC (DS2-like) [8,32] chr. — 5.7 — 15.2 Enc-Dec+CTC (6-1, pretr.)”