Rethinking Evaluation in ASR: Are Our Models Robust Enough?

By Tatiana Likhomanenko, Qiantong Xu, Vineel Pratap, Paden Tomasello, Jacob Kahn, Gilad Avidov, Ronan Collobert, Gabriel Synnaeve. 2020

111 citations recorded by the source corpus.

Is pushing numbers on a single benchmark valuable in automatic speech recognition? Research results in acoustic modeling are typically evaluated based on performance on a single dataset. While the research community has coalesced around various benchmarks, we set out to understand generalization performance in acoustic modeling across datasets -- in particular, if models trained on a single dataset transfer to other (possibly out-of-domain) datasets. Further, we demonstrate that when a large enough set of benchmarks is used, average word error rate (WER) performance over them provides a good proxy for performance on real-world data. Finally, we show that training a single acoustic model on the most widely-used datasets -- combined -- reaches competitive performance on both research and real-world benchmarks.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Safety, Robustness & Interpretability, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Safety, Robustness & Interpretability

In Signal Processing

Citations in the corpus

Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “As demonstrated by Narayanan et al. (2018), Likhomanenko et al. (2020), and Chan et al. (2021) speech recognition systems that are pre-trained in a supervised fashion across many datasets/domains exhibit higher robustness and generalize much more effectively to held-out datasets than models trained…”
SpeechStew: Simply Mix All Available Speech Recognition Data to Train One Large Neural Network (2021, neutral): “Likhomanenko et al. [ 36] also mixed several datasets together to train end-to-end speech recognition models.”
Robust wav2vec 2.0: Analyzing Domain Shift in Self-Supervised Pre-Training (2021, methodology): “It is well known that the performance of ASR systems trained from scratch with conventional supervised objectives can degrade signiﬁcantly when tested on domains mismatched from training data [7, 8].”
Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation (2022, methodology): “We combine S2T datasets from multiple domains to improve the robustness of model training [51, 52], resulting in 196-hr training data for Es-En and 571-hr for En-Es.”
Scaling Speech Technology to 1, 000+ Languages (2023, methodology): “In this section, we turn to building multilingual ASR models on data from multiple domains following a similar approach as prior work for English-only models [Likhomanenko et al., 2020, Chan et al., 2021, Hsu et al., 2021].”
Stabilizing Transformer Training by Preventing Attention Entropy Collapse (2023, methodology): “To speed up the model training (2-3x) and decrease memory usage we are using CAPE positional embedding (Likhomanenko et al., 2021c) instead of relative one (Shaw et al., 2018): both models perform in the same ballpark.”
CAPE: Encoding Relative Positions with Continuous Augmented Positional Embeddings (2021, methodology): “SpecAugment [37] is used as data augmentation in training, and the network architecture follows [30]: the AM encoder is composed of a 1D convolution (kernel 7, stride 3) with a GLU activation and 36 4-heads Transformer layers [48], finally followed by a linear layer which outputs a score for each ta”
SpeechBrain: A General-Purpose Speech Toolkit (2021, result): “We thus focus on more realistic tasks as well, as suggested in some recent works [85, 86].”