Do Deep Convolutional Nets Really Need to be Deep and Convolutional?

By Gregor Urban, Krzysztof J. Geras, Samira Ebrahimi Kahou, Ozlem Aslan, Shengjie Wang, Abdel-rahman Mohamed, Matthai Philipose, Matthew Richardson, R. Caruana. 2016

259 citations recorded by the source corpus.

Subfields: Model Architectures

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Model Architectures

Citations in the corpus

Born Again Neural Networks (2018, background): “Urban et al. (2016) train a super teacher consisting of an ensemble of 16 convolutional neural networks and compresses the learned function into shallow multilayer perceptrons containing 1, 2, 3, 4, and 5 layers.”
FractalNet: Ultra-Deep Neural Networks without Residuals (2016, background): “In the model compression scenario, a deeper network (previously trained) guides and improves the learning of a shallower and faster student network (Ba & Caruana, 2014; Urban et al., 2017).”
FEED: Feature-level Ensemble for Knowledge Distillation (2019, background): “Many studies proposed ideas to transfer knowledge of a teacher to a compact student (Ba and Caruana 2014; Rusu et al. 2015; Bucila, Caruana, and Niculescu-Mizil 2006; Urban et al. 2016).”
Variational Information Distillation for Knowledge Transfer (2019, methodology): “Our method yields a student MLP that significantly outperforms the best-reported MLPs [17, 27] in the literature.”
Model compression via distillation and quantization (2018, methodology): “A model that is too shallow, too narrow, or which misses necessary units, can result in considerable loss of accuracy (Urban et al., 2016).”
Knowledge Distillation: A Survey (2020, background): “Knowledge distillation thus explores the redundancy of parameters in deep model for inference by learning a small student model under the supervision of a large teacher model (Bucilua et al., 2006; Ba and Caruana, 2014; Hinton et al., 2015; Urban et al., 2016), and the key problem of knowledge disti”
N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning (2017, background): “…is the task of training a smaller network (a “student”) to mimic a “teacher” network, performing comparably to the input network (a “teacher”) (Bucilu, Caruana, and Niculescu-Mizil 2006), (Ba and Caruana 2014), (Hinton, Vinyals, and Dean 2015), (Romero et al. 2014), (Urban et al. 2016).”
Improved Knowledge Distillation via Teacher Assistant (2019, background): “Urban et al. (2017) trained a network consisting of an ensemble of 16 convolutional neural networks and compresses the learned function into shallow multilayer perceptrons.”
Training Student Networks for Acceleration with Conditional Adversarial Networks (2018, methodology): “Urban et al. [35] asked the question for convolutional neural networks and claimed the network should at least has a few layers of convolutions.”
Compressing GANs using Knowledge Distillation (2019, background): “, 2019) (Ba & Caurana, 2013) (Urban et al., 2017) and various compression techniques have been exercised to solve this problem (Belagiannis et al.”
ResMLP: Feedforward Networks for Image Classification With Data-Efficient Training (2021, background): “Several works [43], [44], [45] have questioned if FCNs are also competitive on natural image datasets, such as CIFAR10 [27].”
Transparent Model Distillation (2018, background): “Many other papers distilling neural networks into other variants of neural networks followed (Romero et al., 2015; Urban et al., 2017).”