Active Sampler: Light-weight Accelerator for Complex Data Analytics at Scale
By Jinyang Gao, Hosagrahar Visvesvaraya Jagadish, Beng Chin Ooi. 2015
18 citations recorded by the source corpus.
Recent years have witnessed amazing outcomes from "Big Models" trained by "Big Data". Most popular algorithms for model training are iterative. Due to the surging volumes of data, we can usually afford to process only a fraction of the training data in each iteration. Typically, the data are either uniformly sampled or sequentially accessed. In this paper, we study how the data access pattern can affect model training. We propose an Active Sampler algorithm, where training data with more "learning value" to the model are sampled more frequently. The goal is to focus training effort on valuable instances near the classification boundaries, rather than evident cases, noisy data or outliers. We show the correctness and optimality of Active Sampler in theory, and then develop a light-weight vectorized implementation. Active Sampler is orthogonal to most approaches optimizing the efficiency of large-scale data analytics, and can be applied to most analytics models trained by stochastic gradient descent (SGD) algorithm. Extensive experimental evaluations demonstrate that Active Sampler can speed up the training procedure of SVM, feature selection and deep learning, for comparable training quality by 1.6-2.2x.
Subfields: Artificial Intelligence
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Eric Learned-Miller
- Andrew McCallum
- Haw-Shiuan Chang
- Burr Settles
- Yejin Choi
- Percy Liang
- Charles Sutton
- Trevor Cohn
Citations in the corpus
- Active Bias: Training More Accurate Neural Networks by Emphasizing High Variance Samples (2017, methodology): “On the other hand, focusing on harder samples may accelerate and stabilize SGD in cleaner data by minimizing the variance of gradients [1, 12].”
- Tilted Empirical Risk Minimization (2020, unknown): “Such an explicit re-weighting has been explored for other applications (e.g., Chang et al., 2017; Gao et al., 2015; Jiang et al., 2018; Lin et al., 2017; Ren et al., 2018; Shu et al., 2019), though in contrast to these methods, TERM is applicable to a general class of loss functions, with theoretica”