Learning small-size DNN with output-distribution-based criteria

By Jinyu Li, Rui Zhao, Jui-Ting Huang, Yifan Gong. 2014

311 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Model Architectures, Efficiency & Compression, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Model Architectures

In Efficiency & Compression

In Signal Processing

Citations in the corpus

Sequence-Level Knowledge Distillation (2016, methodology): “For classiﬁcation this usually means matching the probabilities either via L 2 on the log scale (Ba and Caruana, 2014) or by cross-entropy (Li et al., 2014; Hinton et al., 2015).”
Distilling the Knowledge in a Neural Network (2015, background): “We have recently become aware of related work on learning a small acoustic model by matching the class probabilities of an already trained larger model [8].”
Contrastive Representation Distillation (2019, methodology): “The method in (Li et al., 2014) was also closely related to (Hinton et al.”
Distilling Knowledge from Deep Networks with Applications to Healthcare Domain (2015, background): “Excellent performance on real world tasks using mimic learning has been recently demonstrated in [29].”
Policy Distillation (2015, methodology): “The contribution of this work is to describe and discuss the policy distillation approach and to demonstrate results on (a) single game distillation, (b) single game distillation with highly compressed models, (c) multi-game distillation, and (d) online distillation.”
Well-Read Students Learn Better: The Impact of Student Initialization on Knowledge Distillation (2019, background): “For instance, distillation for speech recognition is performed on hundreds of millions of data points (Li et al., 2014; Hinton et al., 2015).”
A Teacher-Student Framework for Zero-Resource Neural Machine Translation (2017, background): “Our work is also related to knowledge distillation, which trains a compact model to approximate the function learned by a larger, more complex model or an ensemble of models (Bucila et al., 2006; Ba and Caurana, 2014; Li et al., 2014; Hinton et al., 2015).”
DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantization (2020, methodology): “In the long history of semi-supervised learning (SSL) in speech recognition, self-training approach [2, 3, 4] and knowledge distillation [5], or known as teacher-student model training [6] are the two commonly used SSL methods.”
Label Refinery: Improving ImageNet Classification through Label Progression (2018, background): “Teacher-Student Training: Using another network or an ensemble of multiple networks as a teacher model to train a student model has been explored in [21,22,23,24,25,26].”
Interpretable Deep Models for ICU Outcome Prediction (2016, background): “Furthermore, mimic learning has also been successfully applied to multitask learning, reinforcement learning and speech processing applications 25,26,27 .”
Investigation of transfer learning for ASR using LF-MMI trained neural networks (2017, background): “Frame-level objectives used in model transfer such as using soft-targets [32, 33] are not naively applicable to LF-MMI objective as the neural network outputs are not frame-level posteriors.”
Small-footprint high-performance deep neural network-based speech recognition using split-VQ (2015, background): “Index Terms— DNN, on device speech recognition, model compression, split-VQ”
Knowledge distillation for small-footprint highway networks (2016, methodology): “Instead of manipulating the model parameters, another approach is based on the teacher-student architecture [6, 7, 8], which is also known as model compression [9] or knowledge distillation [10].”
An Investigation of a Knowledge Distillation Method for CTC Acoustic Models (2018, methodology): “KD [18, 19] is a model compression method for DNNs, and often used to bridge the gap of performance between a smaller model and a larger model.”
Cracking the cocktail party problem by multi-beam deep attractor network (2017, methodology): “Wealso built a far-talk ASR model with the domain adaptation method [40] based on teacher-student (T/S) learning [41].”
Knowledge distillation across ensembles of multilingual models for low-resource languages (2017, methodology): “The methodology was modified and applied in [3] to compress a large deep neural network (DNN) into a smaller one so that the latter can be fit into handheld devices without much loss in performance.”
Addressing Missing Labels in Large-Scale Sound Event Recognition Using a Teacher-Student Framework With Loss Masking (2020, background): “ention of ignoring them in the students learning. Note that, unlike other teacher-student pipelines where teachers predictions are used as ground truth to train a student (e.g., via soft labels [17], [18]), our case features a skeptical teacher whose supervision is used to highlight ﬂaws in the curr”
Emotion Recognition in Speech using Cross-Modal Transfer in the Wild (2018, background): “logits [7], or by minimising cross entropy between both models’ probabilistic outputs [43], often through a high-temperature softmax that softens the predictions of each model [19, 34].”