Confident Multiple Choice Learning
By Kimin Lee, Changho Hwang, Kyoungsoo Park, Jinwoo Shin. 2017
52 citations recorded by the source corpus.
Ensemble methods are arguably the most trustworthy techniques for boosting the performance of machine learning models. Popular independent ensembles (IE) relying on naive averaging/voting scheme have been of typical choice for most applications involving deep neural networks, but they do not consider advanced collaboration among ensemble models. In this paper, we propose new ensemble methods specialized for deep neural networks, called confident multiple choice learning (CMCL): it is a variant of multiple choice learning (MCL) via addressing its overconfidence this http URL particular, the proposed major components of CMCL beyond the original MCL scheme are (i) new loss, i.e., confident oracle loss, (ii) new architecture, i.e., feature sharing and (iii) new training method, i.e., stochastic labeling. We demonstrate the effect of CMCL via experiments on the image classification on CIFAR and SVHN, and the foreground-background segmentation on the iCoseg. In particular, CMCL using 5 residual networks provides 14.05% and 6.60% relative reductions in the top-1 error rates from the corresponding IE scheme for the classification task on CIFAR and SVHN, respectively.
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
Citations in the corpus
- Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples (2017, background): “We remark that minimizing a similar KL loss was studied recently for different purposes (Lee et al., 2017; Pereyra et al., 2017).”
- A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks (2018, methodology): “…that the proposed detector using softmax neural classifier trained by standard cross entropy loss typically outperforms the ODIN detector using softmax neural classifier trained by confidence loss [19] which involves jointly training a generator and a classifier to calibrate the posterior distribution.”
- End-to-End Content and Plan Selection for Data-to-Text Generation (2018, methodology): “We further confirm findings by Lee et al. (2017) For all LSTM-based S2S models, we use a two-layer bidirectional LSTM encoder, and hidden and embedding sizes of 750.”
- Towards Oracle Knowledge Distillation with Neural Architecture Search (2019, methodology): “Model ensemble (Breiman 1996; Freund, Schapire, and Abe 1999; Guzman-Rivera, Batra, and Kohli 2012; Lee et al. 2017; 2016; 2015; Mun et al. 2018) is a useful technique to boost performance using multiple models trained ∗Both authors contributed equally Copyright c© 2020, Association for the…”
- A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges (2020, background): “• The results of our research show that although a variety of ensemble methods [474], [475], [476], [477], [478], [479], [480] also have a high ability to deal with uncertainty along with good performance and optimizing the performance of other methods, these high capabilities of these methods have ”
- Residual Mixture of Experts (2022, background): “In the meanwhile, instead of designing experts in the architecture, there are similar research ideas to generate a set of outputs and pick the correct one, such as multi-choice learning [20, 33, 34].”
- Learning to Specialize with Knowledge Distillation for Visual Question Answering (2018, methodology): “As in [19], we share the non-linear activated features right before the first pooling layer, i.”
- Embracing Uncertainty: Decoupling and De-bias for Robust Temporal Grounding (2021, methodology): “Then, most MCL approaches [18, 17, 32] produce the multi-output {fi(x)} N i=1 based on multiple ”base classifiers”, whereas our method generates the multi-output {f(x̂i)} N i=1 via multiple features.”