Curriculum Dropout

By Pietro Morerio, Jacopo Cavazza, Riccardo Volpi, René Vidal, Vittorio Murino. 2017

121 citations recorded by the source corpus.

Dropout is a very effective way of regularizing neural networks. Stochastically “dropping out” units with a certain probability discourages over-specific co-adaptations of feature detectors, preventing overfitting and improving network generalization. Besides, Dropout can be interpreted as an approximate model aggregation technique, where an exponential number of smaller networks are averaged in order to get a more powerful ensemble. In this paper, we show that using a fixed dropout probability during training is a suboptimal choice. We thus propose a time scheduling for the probability of retaining neurons in the network. This induces an adaptive regularization scheme that smoothly increases the difficulty of the optimization problem. This idea of “starting easy” and adaptively increasing the difficulty of the learning problem has its roots in curriculum learning and allows one to train better models. Indeed, we prove that our optimization strategy implements a very general curriculum scheme, by gradually adding noise to both the input and intermediate feature representations within the network architecture. Experiments on seven image classification datasets and different network architectures show that our method, named Curriculum Dropout, frequently yields to better generalization and, at worst, performs just as well as the standard Dropout method.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

When Kernel Methods Meet Feature Learning: Log-Covariance Network for Action Recognition From Skeletal Data (2017, background): “However, the reason for their success depends on the big number of free parameters to optimize, being the latter step complicated (the optimization is non-convex, overfitting is a real issue [24]) and computationally intense (GPU acceleration is fundamental).”
BERT-of-Theseus: Compressing BERT by Progressive Module Replacing (2020, methodology): “Similar to Curriculum Dropout [23], we devise a replacement scheduler to dynamically tune the replacement rate p .”
Curriculum Temperature for Knowledge Distillation (2022, background): “Curriculum Dropout (Morerio et al. 2017) dynamically increases the dropout ratios in order to improve the generalization ability of the model.”
Self-Challenging Improves Cross-Domain Generalization (2020, methodology): “ter that, the new feature tensor Z new is forwarded to the new network output. Finally, the network is updated through back-propagation. Batch Percentage: Some dropout methods like curriculum dropout [17] do not apply dropout at the beginning of training, which improves CNNs by 10 Huang et al. learn”
Curriculum Learning: A Survey (2021, methodology): “ICCV 2017 [44] CV classification PCL dropout continuous model Fan et al.”
Accelerating Training of Transformer-Based Language Models with Progressive Layer Dropping (2020, methodology): “Based on the insights from Section 3.2, and inspired by prior work on curriculum learning [41, 42] we propose a progressive schedule θ ( t ) – a temporal schedule for the expected number of ST blocks that are retained.”
A Survey on Curriculum Learning (2021, methodology): “For example, in Curriculum Dropout [75], the algorithm gradually reduces the ratio of active units in dropout operation from 1 to a predeﬁned θ 0 ∈ (0 , 1) to achieve adaptive regularization during training.”
Geometry Uncertainty Projection Network for Monocular 3D Object Detection (2021, methodology): “So we adopt the widely used polynomial time scheduling function [33] in the curriculum learning topic as our weighted function, which is adapted as follows:”
Adaptive Curriculum Learning (2021, background): “curriculum learning to different research fields including computer vision [1, 9, 23, 30, 33, 36].”
Regularization for Deep Learning: A Taxonomy (2017, background): “Curriculum dropout (Morerio et al., 2017) p(θ|time) Input and hidden features Random Training”