Adaptive Curriculum Learning
By Yajing Kong, Liu Liu, Dacheng Tao. 2021
66 citations recorded by the source corpus.
Inspired by the human learning principle that learning easier concepts first and then gradually paying more attention to harder ones, curriculum learning uses the nonuniform sampling of mini-batches according to the order of examples’ difficulty. Just as a teacher adjusts the curriculum according to the learning progress of each student, a proper curriculum should be adapted to the current state of the model. Therefore, in contrast to recent works using a fixed curriculum, we devise a new curriculum learning method, Adaptive Curriculum Learning (Adaptive CL), adapting the difficulty of examples to the current state of the model. Specifically, we make use of the loss of the current model to adjust the difficulty score while retaining previous useful learned knowledge by KL divergence. Moreover, under a non-linear model and binary classification, we theoretically prove that the expected convergence rate of curriculum learning monotonically decreases with respect to the loss of a point regarding the optimal hypothesis, and monotonically increases with respect to the loss of a point regarding the current hypothesis. The analyses indicate that Adaptive CL could improve the convergence properties during the early stages of learning. Extensive experimental results demonstrate the superiority of the proposed approach over existing competitive curriculum learning methods.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Online Continual Learning with Contrastive Vision Transformer (2022, methodology): “Regularizationbased methods limit the changes in important parameters during the learning of new tasks by estimating the importance of each network parameter for prior tasks [12,42,2,64,43,66,93].”
- Continual Learning through Retrieval and Imagination (2022, background): “Unfortunately, most of the exiting methods fail to satisfy all these guidelines mentioned above (Chaudhry et al. 2021; Wang et al. 2020; Lopez-Paz et al. 2017; Wei et al. 2021; Riemer et al. 2019; Kong et al. 2021; Schwarz et al. 2018; Zhu et al. 2020).”
- Continual Learning with Lifelong Vision Transformer (2022, unknown): “Rehearsal-based methods prevent catastrophic forgetting by replaying a subset of exemplars of previous tasks stored in limited memory [3, 9, 22, 35, 36, 41, 47, 54, 58, 67, 79].”