Balanced Mini-batch Sampling for SGD Using Determinantal Point Processes

By Cheng Zhang, Hedvig Kjellström, Stephan Mandt. 2017

34 citations recorded by the source corpus.

We study a mini-batch diversification scheme for stochastic gradient descent (SGD). While classical SGD relies on uniformly sampling data points to form a mini-batch, we propose a non-uniform sampling scheme based on the Determinantal Point Process (DPP). The DPP relies on a similarity measure between data points and gives low probabilities to mini-batches which contain redundant data, and higher probabilities to mini-batches with more diverse data. This simultaneously balances the data and leads to stochastic gradients with lower variance. We term this approach Diversified Mini-Batch SGD (DM-SGD). We show that regular SGD and a biased version of stratified sampling emerge as special cases. Furthermore, DM-SGD generalizes stratified sampling to cases where no discrete features exist to bin the data into groups. We show experimentally that our method results more interpretable and diverse features in unsupervised setups, and in better classification accuracies in supervised setups.

Subfields: Artificial Intelligence, Statistics and Probability

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Statistics and Probability

Citations in the corpus

Scalable Recollections for Continual Lifelong Learning (2017, background): “One approach to achieving this is a Determinantal Point Process (DPP) (Kulesza et al., 2012) as recently proposed for selecting diverse neural network mini-batches (Zhang et al., 2017).”
Gradient Diversity: a Key Ingredient for Scalable Distributed Learning (2017, methodology): “De et al. (2016) propose an optimization algorithm for choosing the batch-size, and weighted sampling techniques have also been developed (Needell and Ward, 2016, Zhang et al., 2017, Zhao and Zhang, 2014).”
Training Deep Models Faster with Robust, Approximate Importance Sampling (2018, background): “[18, 19] consider sampling diverse minibatches via repulsive point processes.”
Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping (2020, background): “Recently, it has even been shown that that training on mini-batches which are diverse in terms of data or labels (Zhang et al., 2017) can be more sample efﬁcient.”