Testing the correlation of word error rate and perplexity

By Dietrich Klakow, J. Peters. 2002

278 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Speech & Audio

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Speech & Audio

Citations in the corpus

Racial disparities in automated speech recognition (2020, methodology): “We assess the performance of the ASR systems in terms of the word error rate (WER) (27), a standard measure of discrepancy between machine and human transcriptions.”
Ego4D: Around the World in 3,000 Hours of Egocentric Video (2021, methodology): “We adopt the well studied diarization error rate (DER) [11] and word error rate (WER) [114] for di-arization and transcription, respectively.”
BERTScore: Evaluating Text Generation with BERT (2019, background): “PER (Tillmann et al., 1997), WER (Klakow and Peters, 2002), CDER (Leusch et al., 2006), TER (Snover et al., 2006), and TER P (Snover et al., 2009) are metrics based on edit distance.”
Offline recognition of unconstrained handwritten texts using HMMs and statistical language models (2004, background): “Unfortunately, the relationship between the PP of an SLM and the recognition rate of the system using it cannot be modeled clearly [27].”
Polyglot Neural Language Models: A Case Study in Cross-Lingual Phonetic Representation Learning (2016, background): “Perplexity is the standard evaluation measure for language models, which has been shown to correlate strongly with error rates in downstream applications (Klakow and Peters, 2002).”
On Smoothing and Inference for Topic Models (2009, background): “While perplexity is a somewhat indirect measure of predictive performance, it is nonetheless a useful characterization of the predictive qu ality of a language model and has been shown to be wellcorrelated with other measures of performance such worderror rate in speech recognition [Klakow and Peter”
End-to-End Speech Recognition: A Survey (2023, background): “Also, the language model training objective, i.e. language model perplexity, is observed to correlate well with word error rate [311], [312], [313], [314].”
EvalCrafter: Benchmarking and Evaluating Large Video Generation Models (2023, methodology): “Then, we calculate Word Error Rate (WER) [30], Normalized Edit Distance (NED) [51], Character Error Rate (CER) [38], and finally we average these three score to get the OCR-Score.”
D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation (2019, methodology): “The second is unit accuracy, which is metric similar to the word error rate in speech recognition [19].”