Self-Paced Curriculum Learning

By Lu Jiang, Deyu Meng, Qian Zhao, Shiguang Shan, Alexander G. Hauptmann. 2015

591 citations recorded by the source corpus.

Curriculum learning (CL) or self-paced learning (SPL) represents a recently proposed learning regime inspired by the learning process of humans and animals that gradually proceeds from easy to more complex samples in training. The two methods share a similar conceptual learning paradigm, but differ in specific learning schemes. In CL, the curriculum is predetermined by prior knowledge, and remain fixed thereafter. Therefore, this type of method heavily relies on the quality of prior knowledge while ignoring feedback about the learner. In SPL, the curriculum is dynamically determined to adjust to the learning pace of the leaner. However, SPL is unable to deal with prior knowledge, rendering it prone to overfitting. In this paper, we discover the missing link between CL and SPL, and propose a unified framework named self-paced curriculum leaning (SPCL). SPCL is formulated as a concise optimization problem that takes into account both prior knowledge known before training and the learning progress during training. In comparison to human education, SPCL is analogous to "instructor-student-collaborative" learning mode, as opposed to "instructor-driven" in CL or "student-driven" in SPL. Empirically, we show that the advantage of SPCL on two tasks.

Subfields: Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Optimization & Training

Citations in the corpus

Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering (2020, methodology): “, 2009) and SelfPaced Learning (Kumar et al., 2010; Jiang et al., 2015), in which the models are trained with harder Algorithm 1 The Clustering-based Progressive Pretraining”
Listening to Chaotic Whispers: A Deep Learning Framework for News-oriented Stock Trend Prediction (2017, methodology): “In this paper, we choose the linear regularizer proposed in [11], which linearly discriminates samples with respect to their loss, as: Similarto[11][13],weadoptAlternativeConvexSearch(ACS)[9] to solve Equation (1).”
Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, methodology): “, 2015), and this method has proven to be useful in several applications, especially with noisy training data (Jiang et al., 2015).”
Reverse Curriculum Generation for Reinforcement Learning (2017, background): “One particular type of curriculum learning explicitly enables the learner to reject examples which it currently considers too hard [12, 13].”
Learning to Reweight Examples for Robust Deep Learning (2018, background): “Various regularization terms on the example weights have since been proposed to prevent overﬁtting and trivial solutions of assigning weights to be all zeros (Kumar et al., 2010; Ma et al., 2017; Jiang et al., 2015).”
Incremental Learning with Maximum Entropy Regularization: Rethinking Forgetting and Intransigence (2019, methodology): “Further, we employ the idea of curriculum learning (Bengio et al., 2009; Jiang et al., 2015) to further improve the ensembling effect.”
MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, methodology): “After that, many predeﬁned curriculums were proposed, e.g. , in (Supancic & Ramanan, 2013; Jiang et al., 2014; 2015; Sangineto et al., 2016; Chang et al., 2017; Ma et al., 2017a;b).”
Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, background): “Finally, Jiang et al. (2015) explore the idea of self-paced curriculum learning for supervised learning, which unifies and takes advantage of the benefits of self-paced learning and curriculum learning.”
Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks (2018, methodology): “, 2015), or a smaller network that can be quickly trained on the final task? Should we use the same network to continuously rank the training set in order to achieve an adaptive curriculum (as in selfpaced learning: Kumar et al., 2010; Jiang et al., 2015)? In Section 3 we investigate the first two p”
Cost-Effective Active Learning for Deep Image Classification (2017, methodology): “, self-paced learning [27]–[30], we address above-mentioned issues by cost effectively combining the CNN and AL via a complementary sample selection.”
Human-in-the-loop machine learning: a state of the art (2022, methodology): “Nevertheless, some hybrid CL methods are designed which adopt different CL methods making them complement to each other (Jiang et al. 2015; Zhang et al. 2019a).”
A Survey on Curriculum Learning (2021, methodology): “(iii) SPL: if reliable prior knowledge or assumption is available, embedding it into the SPL objective always help [22], [41], [42], [134].”
MentorNet: Regularizing Very Deep Neural Networks on Corrupted Labels (2017, methodology): “As another example, the paper [17] introduced a regularizer G(v) = 12λ ∑n i=1(v 2 i − 2vi).”
On the Convergence and Robustness of Adversarial Training (2021, methodology): “This methodology has been adopted in many applications to enhance model training (Kumar et al., 2010; Jiang et al., 2015).”
What is the Effect of Importance Weighting in Deep Learning? (2018, methodology): “In curriculum learning [Bengio et al., 2009];[Matiisen et al., 2017];[Jiang et al., 2015], training examples are re-weighted by a teacher during the training process with the objective of improving or accelerating training.”
Synthetic vs Real: Deep Learning on Controlled Noise (2019, background): “…2014; Goldberger & Ben-Reuven, 2017; Li et al., 2017b; Veit et al., 2017), example weighting (Jiang et al., 2018; Ren et al., 2018; Shu et al., 2019; Jiang et al., 2015; Liang et al., 2016), semi-supervised learning (Hendrycks et al., 2018; Vahdat, 2017), data augmentation (Zhang et al., 2018;…”
Beyond Synthetic Noise: Deep Learning on Controlled Noisy Labels (2019, methodology): “The regularization term G determines a curriculum (Jiang et al., 2015; 2014; Fan et al., 2017) or equivalently a weighting scheme to compute the latent weight vi to each example.”
Learning the Curriculum with Bayesian Optimization for Task-Specific Word Representation Learning (2016, background): “Curriculum learning and related research on self-paced learning has been explored more deeply in computer vision (Bengio et al., 2009; Kumar et al., 2010; Lee and Grauman, 2011) and in multimedia analysis (Jiang et al., 2015).”
CASSL: Curriculum Accelerated Self-Supervised Learning (2017, methodology): “ion can be hand deﬁned. In Bengio et al. [10], the ranking function is speciﬁed by the variability in object shape. Some other methods like Self-Paced Learning [13] and Self-Paced Curriculum Learning [14] dynamically update the curriculum based on how well the agent is performing. In our method, we ”
A Competence-aware Curriculum for Visual Concepts Learning via Question Answering (2020, methodology): “Self-paced learning [33,27,28,51] quantifies the sample hardness by the training loss and formulates curriculum learning as an optimization problem by jointly modeling the sample selection and the learning objective.”