Unsupervised Training on Large Amounts of Broadcast News Data

By Jeff Z. Ma, Spyros Matsoukas, Owen Kimball, Richard M. Schwartz. 2006

57 citations recorded by the source corpus.

This paper presents our recent effort that aims at improving our Arabic broadcast news (BN) recognition system by using thousands of hours of un-transcribed Arabic audio in the way of unsupervised training. Unsupervised training is first carried out on the 1,900-hour English topic detection and tracking (TDT) data and is compared with the lightly-supervised training method that we have used for the DARPA EARS evaluations. The comparison shows that unsupervised training produces a 21.7% relative reduction in word error rate (WER), which is comparable to the gain obtained with light supervision methods. The same unsupervised training strategy carried out on a similar amount of Arabic BN data produces an 11.6% relative gain. The gain, though considerable, is substantially smaller than what is observed on the English data. Our initial work towards understanding the reasons for this difference is also described

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

Lessons from Building Acoustic Models with a Million Hours of Speech (2019, background): “Semi-supervised learning (SSL) has a long history in ASR ([16], [17], [18]).”
Large scale weakly and semi-supervised learning for low-resource video ASR (2020, methodology): “Sequence-level distillation is the first form of self-labelling applied for speech recognition [20, 9, 27, 5].”
Contrastive Semi-Supervised Learning for ASR (2021, methodology): “Since the 1990s, pseudo-labeling showed improvements to classical HMM/GMM systems as well as hybrid-NN systems [2, 3, 4, 6, 5, 7], and also for end-to-end ASR approaches [8, 9, 10, 11, 12].”
Unsupervised training and directed manual transcription for LVCSR (2010, methodology): “Recently, unsupervised discriminative training has been investigated (Ma et al., 2006; Wang et al., 2007; Yu and Gales, 2007).”
A summary of the 2012 JHU CLSP workshop on zero resource speech technologies and models of early language acquisition (2013, methodology): “Two avenues were explored in the workshop: (i) a multi-lingual corpus was used to train a data-driven, language-invariant front-end for low-resource re ognition; and (ii) untranscribed speech audio was automatically transcribed and used to augment to the labeled data for training, a procedure known ”
Deep neural network features and semi-supervised training for low resource speech recognition (2013, unknown): “Semi-supervised training has been effectively used to train acoustic models in several languages and conditions [32, 33, 34, 35, 36].”
Semi-supervised GMM and DNN acoustic model training with multi-system combination and confidence re-calibration (2013, methodology): “Compared to using a single system recognition result and confidence score, our proposed approach reduces the phone error rate of the inferred transcription by 23.8% relatively when top 60% of data are selected.”
Self-Supervised Speech Representation Learning: A Review (2022, methodology): “relies on a supervised teacher model to label a large volume of speech-only data, which is then used to augment the initial labeled data to train a student model [4], [5], [6], [61].”
Unsupervised versus supervised training of acoustic models (2008, background): “[2, 3] reports the first unsupervised training experiments on very large amounts of English and Arabic broadcast news data.”
Confidence Estimation and Deletion Prediction Using Bidirectional Recurrent Neural Networks (2018, background): “Though schemes such as frame weighting [18, 19] could be used to alleviate the impact of deletions to a certain extent, an explicit deletion prediction is expected to be more appropriate.”
Hubert: How Much Can a Bad Teacher Benefit ASR Pre-Training? (2021, methodology): “Once this encoder is trained and ﬁxed, a BERT transformer model and pre-training loss is applied over latent discrete sequences as a feature extractor for an ASR model [9] or to directly ﬁne-tune the BERT transformer model using a CTC loss for low-resource scenarios [10].”