Earnings-21: A Practical Benchmark for ASR in the Wild

By M. Rio, Natalie Delworth, Ryan Westerman, Michelle Huang, Nishchal Bhandari, Joseph Palakapilly, Quinten McNamara, Joshua Dong, Piotr Żelasko, Miguel Jette. 2021

78 citations recorded by the source corpus.

Commonly used speech corpora inadequately challenge academic and commercial ASR systems. In particular, speech corpora lack metadata needed for detailed analysis and WER measurement. In response, we present Earnings-21, a 39-hour corpus of earnings calls containing entity-dense speech from nine different financial sectors. This corpus is intended to benchmark ASR systems in the wild with special attention towards named entity recognition. We benchmark four commercial ASR models, two internal models built with open-source tools, and an open-source LibriSpeech model and discuss their differences in performance on Earnings-21. Using our recently released fstalign tool, we provide a candid analysis of each model's recognition capabilities under different partitions. Our analysis finds that ASR accuracy for certain NER categories is poor, presenting a significant impediment to transcript comprehension and usage. Earnings-21 bridges academic and commercial ASR system evaluation and enables further research on entity modeling and WER on real world audio.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Speech & Audio, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alec Radford
2. Yoshua Bengio
3. Jong Wook Kim
4. Quoc V. Le
5. Mohit Bansal
6. Luke S. Zettlemoyer
7. Jianfeng Gao
8. Trevor Darrell

In Evaluation & Benchmarks

1. Ilya Sutskever
2. G. Diamos
3. Jong Wook Kim
4. Alec Radford
5. Sergey Levine

In Speech & Audio

1. Ilya Sutskever
2. G. Diamos
3. Greg Brockman
4. Jong Wook Kim
5. Tao Xu
6. Christine McLeavey
7. Alec Radford

In Signal Processing

1. Ilya Sutskever
2. Vijay Janappa Reddi
3. G. Diamos
4. Jong Wook Kim
5. Alec Radford
6. Zhijian Ou
7. Christopher D. Manning
8. Quoc V. Le

Citations in the corpus

• Robust Speech Recognition via Large-Scale Weak Supervision (2022, background): “…in the paper, we use a subset of 25 examples from this data, whose “Ref ID”s are: 2 4 5 8 9 10 12 13 14 16 19 21 23 25 26 28 29 30 33 35 36 37 42 43 45 • Earnings-21 (Del Rio et al., 2021) and Earnings-22 : We used the files available in the speech-datasets repository, as of their 202206 version.”
• The People's Speech: A Large-Scale Diverse English Speech Recognition Dataset for Commercial Usage (2021, background): “Earnings21 [28] is a 39 hour orthographically transcribed speech dataset of public companies’ earning calls created by expert transcriptionists and licensed under a CC-BY-SA license.”