Regularization of Neural Networks using DropConnect

By Li Wan, Matthew D. Zeiler, Sixin Zhang, Yann LeCun, Rob Fergus. 2013

2,649 citations recorded by the source corpus.

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Efficiency & Compression

Citations in the corpus

Deeply-Supervised Nets (2014, methodology): “own on both CIFAR-10 and CIFAR-100 again demonstrates the advantage of the DSN method. 8 Method Error(%) Stochastic Pooling [32] 2.80 Maxout Networks [9] 2.47 Network in Network [20] 2.35 Dropconnect [19] 1.94 DSN (ours) 1.92 Table 3: SVHN classiﬁcation error. 3.3 Street View House Numbers Street Vi”
ImageNet Large Scale Visual Recognition Challenge (2014, methodology): “The network architectures were chosen using the visualization technique of (Zeiler and Fergus, 2013), and they were trained on the GPU following (Zeiler et al., 2011) using the dropout technique (Krizhevsky et al., 2012).”
Network In Network (2013, neutral): “onv. maxout + Dropout [9] 11.68% NIN + Dropout 10.41% CNN + Spearmint + Data Augmentation [13] 9.50% Conv. maxout + Dropout + Data Augmentation [9] 9.38% DropConnect + 12 networks + Data Augmentation [14] 9.32% NIN + Dropout + Data Augmentation 8.81% It turns out in our experiment that using dropout”
Adversarial NLI: A New Benchmark for Natural Language Understanding (2019, background): “Whereas it took around 15 years to achieve “near-human performance” on MNIST (LeCun et al., 1998; Cireşan et al., 2012; Wan et al., 2013) and approximately 7 years to surpass humans on ImageNet (Deng et al.”
MADE: Masked Autoencoder for Distribution Estimation (2015, background): “Speciﬁcally, it bears similarity with the masking in dropconnect networks (Wan et al., 2013).”
Stochastic Optimization of Plain Convolutional Neural Networks with Simple Methods (2020, methodology): “MNIST dataset Lecun et al. (1998) [7] consists of 70,000 28x28 grayscale images of handwritten digits 0 to 9, of which 60,000 are used for training, and 10,000 are used for testing.”
Randaugment: Practical automated data augmentation with a reduced search space (2019, background): “On MNIST, elastic distortions across scale, position, and orientation have been applied to achieve impressive results [43, 4, 49, 42].”
Initialization and Regularization of Factorized Neural Layers (2021, methodology): “For example, SI as deﬁned can be applied to any random initialization, while our FD results suggest that regularizers such as penalty terms and DropConnect (Wan et al., 2013) should be applied on the product matrix rather than directly on individual factors.”
Pay Less Attention with Lightweight and Dynamic Convolutions (2019, methodology): “We found DropConnect to be a good regularizer for the LightConv module (Wan et al., 2013).”
Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation (2020, background): “…we note that models need to be evaluated with the same codebase, training data, and training settings such as learning rate schedule, weight decay, data pre-processing and augmentations, controlling for parameters and FLOPs, architectural regularization [60], training and inference speeds, etc .”
The Goldilocks Principle: Reading Children's Books with Explicit Memory Representations (2015, methodology): “Our use of an ensemble is an alternative way of replicating the application of dropout (Hinton et al., 2012) in the previous best approaches (Hermann et al., 2015) as ensemble averaging has similar effects to dropout (Wan et al., 2013).”