Curriculum Learning: A Survey

By Petru Soviany, Radu Tudor Ionescu, Paolo Rota, Nicu Sebe. 2021

577 citations recorded by the source corpus.

Training machine learning models in a meaningful order, from the easy samples to the hard ones, using curriculum learning can provide performance improvements over the standard training approach based on random data shuffling, without any additional computational costs. Curriculum learning strategies have been successfully employed in all areas of machine learning, in a wide range of tasks. However, the necessity of finding a way to rank the samples from easy to hard, as well as the right pacing function for introducing more difficult data can limit the usage of the curriculum approaches. In this survey, we show how these limits have been tackled in the literature, and we present different curriculum learning instantiations for various tasks in machine learning. We construct a multi-perspective taxonomy of curriculum learning approaches by hand, considering various classification criteria. We further build a hierarchical tree of curriculum learning methods using an agglomerative clustering algorithm, linking the discovered clusters with our taxonomy. At the end, we provide some interesting directions for future work.

Subfields: Artificial Intelligence, Natural Language Processing, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Martin Strobel
2. Bo Dai
3. Hong Chang
4. Xingang Pan
5. Yucheng Lu
6. Christian Theobalt
7. Jianfei Cai
8. Minfeng Xu

In Natural Language Processing

1. Bo Dai
2. Yucheng Lu
3. Christian Theobalt
4. Jianfei Cai
5. Christopher De Sa
6. Dahua Lin
7. Kaixuan Ji
8. Reza Shokri

In Optimization & Training

1. Hong Chang
2. Xingang Pan
3. Bo Dai
4. Yucheng Lu
5. Christian Theobalt
6. Jianfei Cai
7. Minfeng Xu
8. Christopher De Sa

Citations in the corpus

• On The Impact of Machine Learning Randomness on Group Fairness (2023, background): “[52] Petru Soviany, Radu Tudor Ionescu, Paolo Rota, and Nicu Sebe.”
• Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Models (2024, background): “This approach is commonly known as curriculum learning (Bengio et al., 2009; Soviany et al., 2022).”
• Mutual consistency learning for semi-supervised medical image segmentation (2021, background): “Instead of following the curriculum learning (Soviany et al., 2021), our main idea is to exploit the learning for the unlabeled challenging regions to facilitate the model training.”
• GraB: Finding Provably Better Data Permutations than Random Reshuffling (2022, background): “Other approaches, such as curriculum learning [20], try to order the data to mimic human learning and improve generalization [21–23]: these approaches differ from our goal of finding good permutations for minimizing loss in a finite-sum setting.”
• BungeeNeRF: Progressive Neural Radiance Field for Extreme Multi-scale Scene Rendering (2021, background): “Moreover, insights from curriculum learning [7,44,8,5,9,10,33] tell that training models in a meaningful order may ease the training on difficult tasks with model weights initialized on easy samples.”
• Improving Plasticity in Online Continual Learning via Collaborative Learning (2023, background): “2, DC comprises two steps: (1) generating a chain of samples with different levels of difficulty [43] using data augmentation, and (2) distillation of logit distribution from harder samples to easier samples in a collaborative learning way.”