Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks

By Daphna Weinshall, Gad Cohen, Dan Hilman Amir. International Conference on Machine Learning, 2018

295 citations recorded by the source corpus.

Our first contribution in this paper is a theoretical investigation of curriculum learning in the context of stochastic gradient descent when optimizing the least squares loss function. We prove that the rate of convergence of an ideal curriculum learning method in monotonically increasing with the difficulty of the examples, and that this increase in convergence rate is monotonically decreasing as training proceeds. In our second contribution we analyze curriculum learning in the context of training a CNN for image classification. Here one crucial problem is the means to achieve a curriculum. We describe a method which infers the curriculum by way of transfer learning from another network, pre-trained on a different task. While this approach can only approximate the ideal curriculum, we observe empirically similar behavior to the one predicted by the theory, namely, a significant boost in convergence speed at the beginning of training. When the task is made more difficult, improvement in generalization performance is observed. Finally, curriculum learning exhibits robustness against unfavorable conditions such as strong regularization.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/c0877abb93cbe0b28e98929591ec57e7ed213fef

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Bernt Schiele
2. Dacheng Tao
3. Qianru Sun
4. Yaoyao Liu
5. Peter Stone
6. Sergey Levine
7. Jianfeng Gao
8. Mohit Bansal

Citations in the corpus

• Discovering Biased News Articles Leveraging Multiple Human Annotations (2020, methodology): “Our technique is also similar to the stochastic curriculum learning definition (Weinshall et al., 2018), which is a variation of stochastic gradient descent, where the model imports training data instances gradually based on their difficulty score.”
• A Theory for Knowledge Transfer in Continual Learning (2022, background): “Other theoretical frameworks in transfer learning have studied how the degree of relatedness among tasks helps transfer (Lampinen & Ganguli, 2018), and how transfer helps curriculum learning (Weinshall et al., 2018).”
• Few-Shot Text Classification with Triplet Networks, Data Augmentation, and Curriculum Learning (2021, background): “example difficulty exists in the data and then leverage various heuristics to sort examples by difficulty and train models on progressively harder examples (Bengio et al., 2009; Tsvetkov et al., 2016; Weinshall et al., 2018).”
• On The Power of Curriculum Learning in Training Deep Networks (2019, methodology): “As Weinshall et al. (2018) did not employ parameter tuning in their empirical study, the improvement they report might be explained by the use of inappropriate learning rates.”
• Meta-Transfer Learning for Few-Shot Learning (2018, background): “[3] and is popular for multi-task learning [35, 42, 55, 14].”
• Curriculum Learning for Reinforcement Learning Domains: A Framework and Survey (2020, background): “ula are benecial, there is a lack of theoretical results analyzing when and why they are useful, and how they should be created. An initial analysis in the context of supervised learning was done by Weinshall et al. (2018) and Weinshall and Amir (2018). They analyzed whether reordering samples in li”
• Human-in-the-loop machine learning: a state of the art (2022, methodology): “The transfer teacher method is a semi-automatic CL method introduced by Weinshall et al. (2018).”
• Self-Induced Curriculum Learning in Self-Supervised Neural Machine Translation (2020, background): “To overcome the need for labelled data, unsupervised neural machine translation (USNMT) (Lample et al., 2018a; Artetxe et al., 2018b; Yang et al., 2018) focuses on the exploitation of very large amounts of monolingual sentences by combining denoising autoencoders with back-translation and…”
• FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling (2021, methodology): “This technique is widely employed in deep learning research [17–21].”
• Curriculum Learning for Domain Adaptation in Neural Machine Translation (2019, background): “Weinshall and Cohen (2018) provide guidelines for curriculum learning: “A practical curriculum learning method should address two main questions: how to rank the training examples, and how to modify the sampling procedure based on this ranking.””
• Adaptive Curriculum Learning (2021, methodology): “Different from SPL, [11,27,46] obtained the difficulty score, i.”
• Online Adaptative Curriculum Learning for GANs (2018, background): “sampling coefficients to sample data from a buffer. Just like its applications in other parts of machine learning, fundamental aspects of curriculum learning are also being studied (Liu et al., 2017b, Weinshall and Cohen, 2018). While the first work theoretically characterized the convergence rate of ”
• A Survey on Curriculum Learning (2021, methodology): “The teacher model can either be different from the student model and have the greater model capacity (i.e., more complex) [123], or share the same structure with the student model [33], [126].”
• Curriculum Learning: A Survey (2021, methodology): “[105] elaborate an extensive investigation of the behavior of curriculum convolutional models with regard to the difficulty of the samples and the task.”