Diversity Networks

By Zelda E. Mariet, Suvrit Sra. 2015

128 citations recorded by the source corpus.

Abstract: We introduce Divnet, a flexible technique for learning networks with diverse neurons. Divnet models neuronal diversity by placing a Determinantal Point Process (DPP) over neurons in a given layer. It uses this DPP to select a subset of diverse neurons and subsequently fuses the redundant neurons into the selected ones. Compared with previous approaches, Divnet offers a more principled, flexible technique for capturing neuronal diversity and thus implicitly enforcing regularization. This enables effective auto-tuning of network architecture and leads to smaller network sizes without hurting performance. Moreover, through its focus on diversity and neuron fusing, Divnet remains compatible with other procedures that seek to reduce memory footprints of networks. We present experimental results to corroborate our claims: for pruning neural networks, Divnet is seen to be notably superior to competing approaches.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

Citations in the corpus

Data-Efficient Structured Pruning via Submodular Optimization (2022, methodology): “However unlike [Mariet and Sra, 2015], we only use a small fraction of the training data, around ∼ 1% in our experiments.”
The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks (2018, neutral): “Other proposed pruning heuristics include pruning based on activations (Hu et al., 2016), redundancy (Mariet & Sra, 2016; Srinivas & Babu, 2015a), per-layer second derivatives (Dong et al., 2017), and energy/computation efficiency (Yang et al., 2017) (e.g., pruning convolutional filters (Li et al.,…”
Pruning Filters for Efficient ConvNets (2016, neutral): “There has been a signiﬁcant amount of work on reducing the storage and computation costs by model compression (Le Cun et al. (1989); Hassibi & Stork (1993); Srinivas & Babu (2015); Han et al. (2015); Mariet & Sra (2016)).”
What is the State of Neural Network Pruning? (2020, methodology): “Along the same lines, it has been repeatedly shown that, at least for large amounts of pruning, many pruning methods outperform random pruning (Yu et al., 2018; Gale et al., 2019; Frankle et al., 2019; Mariet & Sra, 2015; Suau et al., 2018; He et al., 2017).”
Channel Pruning for Accelerating Very Deep Neural Networks (2017, neutral): “Some works [45, 35, 19] focus on model size compression, which mainly operate the fully connected layers.”
Data-Driven Sparse Structure Selection for Deep Neural Networks (2017, neutral): “On the other hand, the diversity of neurons to be kept is also an important factor to consider [27].”
Designing Energy-Efficient Convolutional Neural Networks Using Energy-Aware Pruning (2016, neutral): “[14] proposed merging filters in the FC layers with similar output activations into one.”
Recent Advances in Convolutional Neural Network Acceleration (2018, unknown): “succeed in merging the qualified neurons with unqualified ones and reduce the network complexity [62].”
MetaPruning: Meta Learning for Automatic Neural Network Channel Pruning (2019, unknown): “Recent works [25, 32, 40, 22, 38, 53] focus on channel pruning in the CNNs, which removes entire weight filters instead of individual weights.”
Proving the Lottery Ticket Hypothesis: Pruning is All You Need (2020, methodology): “Other pruning techniques remove neurons with zero activation [15], or other measures of redundancy [22, 32].”
Improving Adversarial Robustness via Promoting Ensemble Diversity (2019, unknown): “For example, in previous DPP-based methods used to sample diverse subsets (Kwok & Adams, 2012; Mariet & Sra, 2016), they need to calculate the values of det( G ) to obtain the sampling probability for each subset, where m usually scales with the number of classes or number of hidden units in…”
Pruning and Quantization for Deep Neural Network Acceleration: A Survey (2021, methodology): “Comparing with the similarity based approach above, diversity networks (DIVNET) [157] consider the calculation redundancy based on the activations.”
The Lottery Ticket Hypothesis: Training Pruned Neural Networks (2018, unknown): “ connections to increase network capacity after small weights have been pruned and surviving weights ﬁne-tuned. Other proposed pruning heuristics include pruning based on activations [24], redundancy [37, 44], perlayer second derivatives [9], and energy/computation efﬁciency [50] (e.g., pruning conv”
On Compressing Deep Models by Low Rank and Sparse Decomposition (2017, unknown): “Mariet and Sra [19] sampled the most uncorrelated neurons by the determinant point processes, and removed other highly correlated neurons.”