Blending LSTMs into CNNs

By Krzysztof J. Geras, Abdel-rahman Mohamed, R. Caruana, Gregor Urban, Shengjie Wang, Ozlem Aslan, Matthai Philipose, Matthew Richardson, Charles Sutton. 2015

61 citations recorded by the source corpus.

We consider whether deep convolutional networks (CNNs) can represent decision functions with similar accuracy as recurrent networks such as LSTMs. First, we show that a deep CNN with an architecture inspired by the models recently introduced in image recognition can yield better accuracy than previous convolutional and LSTM networks on the standard 309h Switchboard automatic speech recognition task. Then we show that even more accurate CNNs can be trained under the guidance of LSTMs using a variant of model compression, which we call model blending because the teacher and student models are similar in complexity but different in inductive bias. Blending further improves the accuracy of our CNN, yielding a computationally efficient model of accuracy higher than any of the other individual models. Examining the effect of "dark knowledge" in this model compression task, we find that less than 1% of the highest probability labels are needed for accurate model compression.

Subfields: Artificial Intelligence, Signal Processing

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Signal Processing

Citations in the corpus

FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “Geras (Geras et al. 2015) try to transfer knowledge between two networks that have almost the same number of parameters.”
Towards Understanding Knowledge Distillation (2019, background): “It has been found to work well across a wide range of applications, including e.g. transferring from one architecture to another (Geras et al., 2016), compression (Howard et al., 2017; Polino et al., 2018), integration with ﬁrst-order logic (Hu et al., 2016) or other prior knowledge (Yu et al.,…”
A deep learning approach for real-time detection of atrial fibrillation (2019, background): “Therefore, a combination of these two networks achieves a higher classification accuracy (Geras et al., 2015; Shi et al., 2015; Sainath et al., 2015).”
Distillation-Based Training for Multi-Exit Architectures (2019, methodology): “Since then, the technique has been found to beneﬁt a wide array of applications, including transferring from one architecture to another [12], compression [17, 32], integration with ﬁrst-order logic [18] or other prior knowledge [47], learning from noisy labels [26], defending against adversarial…”
Learning Approximate Inference Networks for Structured Prediction (2018, background): “Geras et al. (2016) train a convolutional network to mimic an LSTM for speech recognition.”
Feature-Level Ensemble Knowledge Distillation for Aggregating Knowledge from Multiple Networks (2020, background): “Geras [8] try to transfer knowledge between two networks that have almost the same number of parameters.”
Do Deep Convolutional Nets Really Need to be Deep and Convolutional? (2016, methodology): “Similarly, Seide et al. (2011) and Geras et al. (2015) show that deeper models are more accurate than shallow models in speech acoustic modeling.”
Efficient Knowledge Distillation from an Ensemble of Teachers (2017, methodology): “In the knowledge distillation framework, instead of training models which had reduced computational requirements and improved latency performances directly on hard targets in a single step, training is now performed in two separate steps [4, 5, 6, 7, 8, 9, 10].”
Knowledge distillation across ensembles of multilingual models for low-resource languages (2017, methodology): “[6] investigated the number of labels and the interpolation weight of teacher’s labels with original labels.”