Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networks

By Anton Ragni, Qiujia Li, Mark John Francis Gales, Yu Wang. 2018

33 citations recorded by the source corpus.

The standard approach to assess reliability of automatic speech transcriptions is through the use of confidence scores. If accurate, these scores provide a flexible mechanism to flag transcription errors for upstream and downstream applications. One challenging type of errors that recognisers make are deletions. These errors are not accounted for by the standard confidence estimation schemes and are hard to rectify in the upstream and downstream processing. High deletion rates are prominent in limited resource and highly mismatched training/testing conditions studied under IARPA Babel and Material programs. This paper looks at the use of bidirectional recurrent neural networks to yield confidence estimates in predicted as well as deleted words. Several simple schemes are examined for combination. To assess usefulness of this approach, the combined confidence score is examined for untranscribed data selection that favours transcriptions with lower deletion errors. Experiments are conducted using IARPA Babel/Material program languages.

Subfields: Artificial Intelligence, Software, Natural Language Processing, Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Software

In Natural Language Processing

In Model Architectures

Citations in the corpus

XTREME-S: Evaluating Cross-lingual Speech Representations (2022, background): “Recent work revived this dataset with different preprocessing [16, 17, 18, 4].”
Addressing Failure Prediction by Learning Model Confidence (2019, background): “Similarly, [42, 29] mention the use of bi-directional lattice RNN specifically designed for confidence estimation in speech recognition, whereas ConfidNet offers a model- and task-agnostic approach which can be plugged into any deep neural network.”
Uncertainty Estimation in Autoregressive Structured Prediction (2021, methodology): “Previous work has examined bespoke supervised confidence estimation techniques for each task separately (Evermann & Woodland, 2000; Liao & Gales, 2007; Ragni et al., 2018; Chen et al., 2017; Koehn, 2009; Kumar & Sarawagi, 2019) which construct an "error-detection" model on top of the original ASR/NM”