Stochastic Multiple Choice Learning for Training Diverse Deep Ensembles

By Stefan Lee, Senthil Purushwalkam, Michael Cogswell, Viresh Ranjan, David J. Crandall, Dhruv Batra. 2016

199 citations recorded by the source corpus.

Many practical perception systems exist within larger processes that include interactions with users or additional components capable of evaluating the quality of predicted solutions. In these contexts, it is beneficial to provide these oracle mechanisms with multiple highly likely hypotheses rather than a single prediction. In this work, we pose the task of producing multiple outputs as a learning problem over an ensemble of deep networks -- introducing a novel stochastic gradient descent based approach to minimize the loss with respect to an oracle. Our method is simple to implement, agnostic to both architecture and loss function, and parameter-free. Our approach achieves lower oracle error compared to existing methods on a wide range of tasks and deep architectures. We also show qualitatively that the diverse solutions produced often provide interpretable representations of task ambiguity.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

Citations in the corpus

Learning Independent Causal Mechanisms (2017, methodology): “Lee et al. (2016) propose Stochastic Multiple Choice Learning, an algorithm which resembles the one we describe in Section 3, aimed at training mixture of experts to propose a diverse set of outputs.”
Mixture Models for Diverse Machine Translation: Tricks of the Trade (2019, methodology): “In the online EM algorithm, we minimize the loss via stochastic gradient descent, effectively interleaving the E-step and Mstep for each mini-batch (Lee et al., 2016).”
End-to-End Content and Plan Selection for Data-to-Text Generation (2018, background): “To maximize the target, Guzman-Rivera et al. 2016) reduce the computational overhead by introducing a stochastic MCL (sMCL) variant that does not require retraining.”
MoViNets: Mobile Video Networks for Efficient Video Recognition (2021, background): “More recent results indicate that deep ensembles of small models can be more efficient than single large models on image classification [33, 44, 56, 35, 21], and we extend these findings to video classification.”
Ensemble deep learning in bioinformatics (2020, background): “An alternative approach to increase base model diversity is through multiple choice learning in which each network is ‘specialized’ on a particular subset of data during the training step 26 .”
Transformation-Grounded Image Generation Network for Novel 3D View Synthesis (2017, background): “We believe the reason is the inherently deterministic nature of our encoder-decoder architecture, which can be alleviated by incorporating approaches like explicit diverse training [27] or probabilistic generative modeling [47, 48, 29, 43].”
Towards Oracle Knowledge Distillation with Neural Architecture Search (2019, methodology): “Model ensemble (Breiman 1996; Freund, Schapire, and Abe 1999; Guzman-Rivera, Batra, and Kohli 2012; Lee et al. 2017; 2016; 2015; Mun et al. 2018) is a useful technique to boost performance using multiple models trained ∗Both authors contributed equally Copyright c© 2020, Association for the…”
Confident Multiple Choice Learning (2017, methodology): “ating a set of plausible outputs such that at least one of them is correct for the second stage (b), e.g., human operators. Under this motivation, MCL has been studied (Guzman-Rivera et al.,2014;2012;Lee et al., 2016), where various applications have been demonstrated, e.g., image classiﬁcation (Kri”
Learning to Specialize with Knowledge Distillation for Visual Question Answering (2018, background): “The issue has not been addressed in the MCL approaches as they mainly focus on simple image classification tasks with at most 10 labels [10, 19, 21].”
Query-Centric Trajectory Prediction (2023, methodology): “On the other hand, we adopt the winnertake-all strategy [29] to optimize the locations and scales output by the proposal and refinement modules, which conducts backpropagation on the best-predicted proposal and its refinement only.”
On building ensembles of stacked denoising auto-encoding classifiers and their further improvement (2018, background): “We conclude this concise review mentioning two very recent contributions: Simultaneously training aggregation weights and learners for several DNN architectures, including CNNs [63] , and selectively combining CNNs that have been trained with powerful augmentation procedures [28] .”
Deepdiary: Lifelogging image captioning and summarization (2018, background): “There is also recent work [56] aimed at training deep networks to produce diverse outputs by using an ensemble set of predictors which capture different data distributions.”
A Probabilistic U-Net for Segmentation of Ambiguous Images (2018, methodology): “(2)In [1] this is defined as ambiguous evidence in contrast to implicit class confusion, that stems from an ambiguous class definition (e.”
Multimodal Trajectory Predictions for Autonomous Driving using Deep Convolutional Networks (2018, unknown): “To overcome this problem researchers proposed training an ensemble of networks [36], or a single network to produce M different outputs for M different hypotheses [37] using a loss that only accounts for the closest prediction to ground truth labels.”
Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles (2016, background): “Explicitly de-correlating networks’ predictions, e.g. as in (Lee et al., 2016), might promote ensemble diversity and improve performance even further.”
Residual Mixture of Experts (2022, background): “In the meanwhile, instead of designing experts in the architecture, there are similar research ideas to generate a set of outputs and pick the correct one, such as multi-choice learning [20, 33, 34].”
Interactive Image Segmentation with Latent Diversity (2018, background): “The recent work of Lee et al. [27] applies these ideas to deep networks, but considers an ensemble of networks and modiﬁes the learning algorithm to selectively channel the gradient ﬂow through individual networks in the ensemble.”
Co-regularized Alignment for Unsupervised Domain Adaptation (2018, background): “Several ensemble methods also encourage diversity among the classifiers in the ensemble [25, 23].”