Blockout: Dynamic Model Selection for Hierarchical Deep Networks

By Calvin Murdock, Z. Li, Howard Zhou, Tom Duerig. 2015

52 citations recorded by the source corpus.

Most deep architectures for image classification-even those that are trained to classify a large number of diverse categories-learn shared image representations with a single model. Intuitively, however, categories that are more similar should share more information than those that are very different. While hierarchical deep networks address this problem by learning separate features for subsets of related categories, current implementations require simplified models using fixed architectures specified via heuristic clustering methods. Instead, we propose Blockout, a method for regularization and model selection that simultaneously learns both the model architecture and parameters. A generalization of Dropout, our approach gives a novel parametrization of hierarchical architectures that allows for structure learning via back-propagation. To demonstrate its utility, we evaluate Blockout on the CIFAR and Image Net datasets, demonstrating improved classification accuracy, better regularization performance, faster training, and the clear emergence of hierarchical network structures.

Subfields: Artificial Intelligence, Computer Networks and Communications

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

Citations in the corpus

N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning (2017, background): “6), Baker et al. (2016), Ludermir et al. (2006), Miikkulainen et al. (2017), Real et al. (2017), Snoek et al. (2012), Snoek et al. (2015), Stanley & Miikkulainen (2002), Jozefowicz et al. (2015), Murdock et al. (2016), Feng & Darrell (2015), Warde-Farley et al. (2014), Iandola et al. (2016).”
Random Erasing Data Augmentation (2017, methodology): “(Murdock et al. 2016) propose “Blockout”, a method for simultaneous regularization and model selection via masked weight matrices on CNN layers.”
SplitNet: Learning to Semantically Split Deep Networks for Parameter Reduction and Model Parallelization (2017, background): “Murdock et al. (2016) generalized dropout to stochastically assign nodes to clusters which results in obtaining a hierarchical structure.”
CNN-RNN: a large-scale hierarchical image classification framework (2017, methodology): “Instead of utilizing a fixed architecture for classification, Mur-dock et al. [29] proposed a regularization method, i.e. Blockout, to automatically learn the hierarchical structure.”
3D Point Cloud Generative Adversarial Network Based on Tree Structured Graph Convolutions (2019, background): “Tree-structured Deep Networks: There have been several attempts to represent convolutional neural networks or long short-term memory using tree structures [5, 20, 27, 28, 31].”
Deep learning for visual understanding (2017, unknown): “[275] proposed a regularization method, i.”
NBDT: Neural-Backed Decision Trees (2020, methodology): “p to datasets like CIFAR10[25], CIFAR100[25], TinyImageNet[26], and ImageNet[13]. One category of models organizes the neural network into a hierarchy, dynamically selecting branches to run inference [47, 28, 46, 35, 31]. However, these models forsake interpretability by creating impure leaves. Othe”
Deep Multi-Modal Object Detection and Semantic Segmentation for Autonomous Driving: Datasets, Methods, and Challenges (2019, methodology): “Another way to optimize the network structure is by regularization, such as l1 regularization [230] and stochastic regularization [231], [232].”
Deep Multimodal Learning: A Survey on Recent Advances and Trends (2017, methodology): “Blockout [88] can perform simultaneous regularization and model selection through a clever technique that stochastically assigns hidden units to “clusters,” forming block-structured weight matrices.”
Attentive Single-Tasking of Multiple Tasks (2019, unknown): “For multi-tasking, Cross Stitch Networks [39] automatically learn to split/fuse two independent networks in different depths according to their learned tasks, while [41] estimate a block-structured weight matrix during CNN training.”