What Objective Does Self-paced Learning Indeed Optimize?

By Deyu Meng, Qian Zhao. 2015

80 citations recorded by the source corpus.

Self-paced learning (SPL) is a recently raised methodology designed through simulating the learning principle of humans/animals. A variety of SPL realization schemes have been designed for different computer vision and pattern recognition tasks, and empirically substantiated to be effective in these applications. However, the investigation on its theoretical insight is still a blank. To this issue, this study attempts to provide some new theoretical understanding under the SPL scheme. Specifically, we prove that the solving strategy on SPL accords with a majorization minimization algorithm implemented on a latent objective function. Furthermore, we find that the loss function contained in this latent objective has a similar configuration with non-convex regularized penalty (NSPR) known in statistics and machine learning. Such connection inspires us discovering more intrinsic relationship between SPL regimes and NSPR forms, like SCAD, LOG and EXP. The robustness insight under SPL can then be finely explained. We also analyze the capability of SPL on its easy loss prior embedding property, and provide an insightful interpretation to the effectiveness mechanism under previous SPL variations. Besides, we design a group-partial-order loss prior, which is especially useful to weakly labeled large-scale data processing tasks. Through applying SPL with this loss prior to the FCVID dataset, which is currently one of the biggest manually annotated video dataset, our method achieves state-of-the-art performance beyond previous methods, which further helps supports the proposed theoretical arguments.

Subfields: Artificial Intelligence, Computational Mechanics

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Lu Jiang
2. Li-Jia Li
3. Deyu Meng
4. Yueting Zhuang
5. Li Fei-Fei
6. Zongben Xu
7. Jun Shu
8. Eric P. Xing

In Computational Mechanics

1. Eric Learned-Miller
2. Andrew McCallum
3. Haw-Shiuan Chang
4. Jun Shu
5. Jun Xiao
6. Zhongfei Zhang
7. Deyu Meng
8. Xi Li

Citations in the corpus

• Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, methodology): “By decreasing the weight of difficult examples in the loss function, the model may become more robust to outliers [33], and this method has proven useful in several applications, especially with noisy labels [36].”
• MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels (2017, background): “Meng et al. (2015) investigated the insights of self-paced objective function, and proved that the optimization of SPL algorithm is intrinsically equivalent to minimizing a robust loss function.”
• Small Sample Learning in Big Data Era (2018, background): “Recently, Meng et al. (2017) demonstrated an insightful understanding that self-paced learning is robustness to outliers/heavy noises account for learn with a latent non-convex regularized penalty.”
• MentorNet: Regularizing Very Deep Neural Networks on Corrupted Labels (2017, background): “This may have profound impacts on many vision applications [28, 9, 21, 38, 16, 32].”
• Self-Paced Boost Learning for Classification (2016, background): “U 2 Rn⇥C is the Lagrangian multiplier of the equality constraints of Eq.”