WER we are and WER we think we are

By Piotr Szyma'nski, Piotr Żelasko, Mikolaj Morzy, Adrian Szymczak, Marzena Zyla-Hoppe, Joanna Banaszczak, Lukasz Augustyniak, Jan Mizgajski, Yishay Carmiel. 2020

54 citations recorded by the source corpus.

Natural language processing of conversational speech requires the availability of high-quality transcripts. In this paper, we express our skepticism towards the recent reports of very low Word Error Rates (WERs) achieved by modern Automatic Speech Recognition (ASR) systems on benchmark datasets. We outline several problems with popular benchmarks and compare three state-of-the-art commercial ASR systems on an internal dataset of real-life spontaneous human conversations and HUB’05 public benchmark. We show that WERs are significantly higher than the best reported results. We formulate a set of guidelines which may aid in the creation of real-life, multi-domain datasets with high quality annotations for training and testing of robust ASR systems.

Subfields: Artificial Intelligence, Evaluation & Benchmarks, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Evaluation & Benchmarks

In Speech & Audio

Citations in the corpus

Rethinking Evaluation in ASR: Are Our Models Robust Enough? (2020, background): “Very recently, authors of [6] point out some limitations of current ASR benchmarks, and propose guidelines to create multi-domain datasets.”
SpeechBrain: A General-Purpose Speech Toolkit (2021, result): “We thus focus on more realistic tasks as well, as suggested in some recent works [85, 86].”