On The Power of Curriculum Learning in Training Deep Networks

By Guy Hacohen, Daphna Weinshall. International Conference on Machine Learning, 2019

578 citations recorded by the source corpus.

Training neural networks is traditionally done by providing a sequence of random mini-batches sampled uniformly from the entire training data. In this work, we analyze the effect of curriculum learning, which involves the non-uniform sampling of mini-batches, on the training of deep networks, and specifically CNNs trained for image recognition. To employ curriculum learning, the training algorithm must resolve 2 problems: (i) sort the training examples by difficulty; (ii) compute a series of mini-batches that exhibit an increasing level of difficulty. We address challenge (i) using two methods: transfer learning from some competitive ``teacher" network, and bootstrapping. In our empirical evaluation, both methods show similar benefits in terms of increased learning speed and improved final performance on test data. We address challenge (ii) by investigating different pacing functions to guide the sampling. The empirical investigation includes a variety of network architectures, using images from CIFAR-10, CIFAR-100 and subsets of ImageNet. We conclude with a novel theoretical analysis of curriculum learning, where we show how it effectively modifies the optimization landscape. We then define the concept of an ideal curriculum, and show that under mild conditions it does not change the corresponding global minimum of the optimization function.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

View Paper: https://www.semanticscholar.org/paper/ba013acca784bfea74241c879e18b72984950493

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Dacheng Tao
2. Yanjun Qi
3. Jiawei Han
4. Yu Meng
5. Liu Liu
6. Paul N. Bennett
7. Yajing Kong
8. Xia Song

In Computer Vision and Pattern Recognition

1. Dacheng Tao
2. Jingkuan Song
3. Luc Van Gool
4. Lianli Gao
5. Yi Yang
6. Trevor Darrell
7. Ming-Hsuan Yang
8. Liu Liu

Citations in the corpus

• Manipulating SGD with Data Ordering Attacks (2021, background): “[13]) is particularly close to our batch reordering setting.”
• Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, background): “…sentence representations obtained from NMT systems or tailored architectures are achieving a new state-of-the-art in parallel sentence extraction and filtering (Espa˜na-Bonet et al., 2017; Gr´egoire and Langlais, 2018; Artetxe and Schwenk, 2019; Hangya and Fraser, 2019; Chaudhary et al., 2019).”
• ClipGen: A Deep Generative Model for Clipart Vectorization and Synthesis (2021, methodology): “Inspired by the curriculum learning training strategy used in [63], [64], we categorized the entire training process of our generative model into”
• Human-in-the-loop machine learning: a state of the art (2022, background): “Hacohen and Weinshall (2019) demonstrated that transfer teacher is the most robust automatic CL, and that the advantage of CL is more effective when the task is difficult.”
• Pretraining Text Encoders with Adversarial Mixture of Training Signal Generators (2022, background): “In general, configuring a learning curriculum for neural model training is tricky (Bengio et al., 2009; Hacohen & Weinshall, 2019).”
• Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learning (2020, background): “Prior work has shown that training with a curriculum improves performance in several machine learning tasks (Bengio et al. 2009; Hacohen and Weinshall 2019).”
• Curriculum-Based Meta-learning (2021, background): “In the family of curriculum learning [5, 13], the idea of learning from Poster Session 2 MM ’21, October 20–24, 2021, Virtual Event, China”
• Adaptive Curriculum Learning (2021, methodology): “Different from SPL, [11,27,46] obtained the difficulty score, i.”
• CILEA-NET: Curriculum-Based Incremental Learning Framework for Remote Sensing Image Classification (2021, unknown): “Approaches like active learning are similar to the curriculum learning paradigm but essentially differ from it due to the dynamic sampling of training points based on the current hypothesis of the model [36].”