Scaling Large Learning Problems with Hard Parallel Mixtures

By Ronan Collobert, Yoshua Bengio, Samy Bengio. 2002

50 citations recorded by the source corpus.

A challenge for statistical learning is to deal with large data sets, e.g. in data mining. Popular learning algorithms such as Support Vector Machines have training time at least quadratic in the number of examples: they are hopeless to solve problems with a million examples. We propose a "hard parallelizable mixture" methodology which yields significantly reduced training time through modularization and parallelization: the training data is iteratively partitioned by a "gater" model in such a way that it becomes easy to learn an "expert" model separately in each region of the partition. Ap robabilistic extension and the use of a set of generative models allows representing the gater so that all pieces of the model are locally trained. For SVMs, time complexity appears empirically to locally grow linearly with the number of examples, while generalization performance can be enhanced. For the probabilistic version of the algorithm, the iterative algorithm provably goes down in a cost function that is an upper bound on the negative log-likelihood.

Subfields: Artificial Intelligence, Statistical and Nonlinear Physics, Model Architectures, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistical and Nonlinear Physics

In Model Architectures

In Optimization & Training

Citations in the corpus

Unified Scaling Laws for Routed Language Models (2022, background): “Many sparsely-activated networks have this property, but here we exclusively study the layout based on Sparse Mixtures of Experts [Shazeer et al., 2017] where multiple sub-components of a deep neural network (i.e., several layers) are independently converted to routed equivalents and jointly trained”
Learning Factored Representations in a Deep Mixture of Experts (2013, methodology): “A method that accomplishes this for a single layer has been described by Collobert et al. [4], which could possibly be adapted to our multilayer case; we hope to address this in future work.”
Deep Learning of Representations: Looking Forward (2013, background): “, the hard mixture of experts we introduced a decade ago (Collobert et al., 2003)) is that the gating units should not be mutually exclusive and should instead form a distributed pattern.”
Hash Layers For Large Sparse Models (2021, background): “Several works have shown that sparsity can be maintained during training, e.g. [9, 7, 8, 10].”
Hard Mixtures of Experts for Large Scale Weakly Supervised Vision (2017, neutral): “This is very similar to the approach in [3].”
Deep Mixture of Experts via Shallow Embedding (2018, background): “Later work [7] generalized this mixture of experts formulation to a non-probabilistic setting where the gating function G outputs arbitrary weights for the experts instead of probabilities.”
Learning to Compose Topic-Aware Mixture of Experts for Zero-Shot Video Captioning (2018, methodology): “Then MoE has been applied to various machine learning algorithms (Jordan and Jacobs 1994; Collobert, Bengio, and Bengio 2003), such as SVMs (Collobert, Bengio, and Bengio 2002), Gaussian Processes (Tresp 2001), and deep networks (Ahmed, Baig, and Torresani 2016; Gu et al. 2018).”
Learning Layer-Skippable Inference Network (2020, methodology): “A Mixture of Experts [42], [43] (MoE) consists of experts and a gating network.”
A Constructive Hybrid Structure Optimization Methodology for Radial Basis Probabilistic Neural Networks (2008, methodology): “As usually done in the literature [29], [ 30 ], it was modified as a binary classification problem where the goal was to separate class 2 from the other six classes.”