Hallucination of Speech Recognition Errors With Sequence to Sequence Learning

By Prashant Serai, Vishal Sunder, Eric Fosler-Lussier. 2021

24 citations recorded by the source corpus.

Prior work in this domain has focused on modeling errors at the phonetic level, while using a lexicon to convert the phones to words, usually accompanied by an FST Language model. We present novel end-to-end models to directly predict hallucinated ASR word sequence outputs, conditioning on an input word sequence as well as a corresponding phoneme sequence. This improves prior published results for recall of errors from an in-domain ASR system’s transcription of unseen data, as well as an out-of-domain ASR system’s transcriptions of audio from an unrelated task, while additionally exploring an in-between scenario when limited characterization data from the test ASR system is obtainable. To verify the extrinsic validity of the method, we also use our hallucinated ASR errors to augment training for a spoken question classifier, finding that they enable robustness to real ASR errors in a downstream task, when scarce or even zero task-specific audio was available at train-time.

Subfields: Artificial Intelligence, Natural Language Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Pascale Fung
2. Andrea Madotto
3. Wenliang Dai
4. Nayeon Lee
5. Yan Xu
6. Etsuko Ishii
7. Ziwei Ji
8. Dan Su

In Natural Language Processing

1. Pascale Fung
2. Nayeon Lee
3. Andrea Madotto
4. Etsuko Ishii
5. Yejin Bang
6. Yan Xu
7. Rita Frieske
8. Dan Su

Citations in the corpus

• Survey of Hallucination in Natural Language Generation (2022, background): “We also briely summarize hallucinations in multi-modal settings such as visual-language tasks [1, 10] and speech-to-text tasks [119, 124].”