Semi-supervised training of Deep Neural Networks

By Karel Veselý, Mirko Hannemann, Lukáš Burget. 2013

156 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Representation & Self-Supervised Learning

Citations in the corpus

Effectiveness of self-supervised pre-training for speech recognition (2019, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [5, 6, 7, 8, 9, 10, 11, 12], in addition to semi- and weakly-supervised learning techniques to reduce the supervised data”
Self-Training for End-to-End Speech Recognition (2019, methodology): “Training with pseudo-labels can give an improvement to WER not only for low-resource languages [6, 7] but also on large-scale data sets [21].”
Deep Learning: Methods and Applications (2014, methodology): “[374], where self-training strategies are used as the basis for data selection using both the utterance-level and frame-level confidences.”
Deep Contextualized Acoustic Representations for Semi-Supervised Speech Recognition (2019, background): “Other SSL approaches include: pre-training on a large amount of unlabeled data with restricted Boltzmann machines (RBMs) [6]; entropy minimization [7, 8, 9], where the uncertainty of the unlabeled data is incorporated as part of the training objective; and graph-based approaches [10], where the…”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “In the long history of semi-supervised learning (SSL) in speech recognition, self-training approach [2, 3, 4] and knowledge distillation [5], or known as teacher-student model training [6] are the two commonly used SSL methods.”
Semi-Supervised and Cross-Lingual Knowledge Transfer Learnings for DNN Hybrid Acoustic Models Under Low-Resource Conditions (2016, methodology): “Technically, this is similar to the frame based data selection method proposed in [2] since we use the same method advocated in [18] to estimate word confidence and time boundaries in Kaldi(3).”
Semi-Supervised DNN Training with Word Selection for ASR (2017, methodology): “We previously compared the sentence-selection with frameselection in [10].”
Semi-supervised ensemble DNN acoustic model training (2017, methodology): “When we have a frame-level CMS, it is possible to perform frame-level data selection (frame dropping [7]) in the fine-tuning step of DNN over frame-level mini-batches.”
Effectiveness of Self-Supervised Pre-Training for ASR (2020, background): “The practical need for building ASR systems for new conditions with limited resources spurred a lot of work focused on unsupervised speech recognition and representation learning [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14], in addition to semi- and weakly-supervised learning techniques to reduce th”
Semi-Supervised Training of Acoustic Models Using Lattice-Free MMI (2018, methodology): “Confidence scores can be at the frame level [6], word level [7] or utterance level [6, 8, 9].”
Semi-supervised maximum mutual information training of deep neural network acoustic models (2015, methodology): “In addition, filtering of untranscribed data frames using say, frame-level confidence [11], can be incorporated easily.”
Almost-unsupervised Speech Recognition with Close-to-zero Resource Based on Phonetic Structures Learned from Very Small Unpaired Speech and Text Data (2018, background): “Although substantial efforts have been reported on semi-supervised ASR [1, 2, 3, 4, 5, 6], in most cases a large amount of speech data including a good portion annotated were still needed.”
Towards Unsupervised Learning of Speech Features in the Wild (2021, unknown): “Most papers used conﬁdence-based ﬁltering [12, 13, 14, 15].”