Large-Scale Learning with Less RAM via Randomization

By D. Golovin, David Sculley, H. Brendan McMahan, Michael Young. 2013

25 citations recorded by the source corpus.

We reduce the memory footprint of popular large-scale online learning methods by projecting our weight vector onto a coarse discrete set using randomized rounding. Compared to standard 32-bit float encodings, this reduces RAM usage by more than 50% during training and by up to 95% when making predictions from a fixed model, with almost no loss in accuracy. We also show that randomized counting can be used to implement percoordinate learning rates, improving model quality with little additional RAM. We prove these memory-saving methods achieve regret guarantees similar to their exact variants. Empirical evaluation confirms excellent performance, dominating standard approaches across memory versus accuracy tradeoffs.

Subfields: Artificial Intelligence, Management Science and Operations Research, Optimization & Training

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Management Science and Operations Research

In Optimization & Training

Citations in the corpus

Federated Learning: Strategies for Improving Communication Efficiency (2016, background): “Incremental, randomized and distributed optimization algorithms can be similarly analyzed in a quantized updates setting [16, 8, 7].”
Stochastic, Distributed and Federated Optimization for Machine Learning (2017, background): “Incremental, randomized and distributed optimization algorithms can be similarly analyzed in a quantized updates setting [144, 66, 64].”
A Provable Algorithm for Learning Interpretable Scoring Systems (2018, methodology): “com/software method (see [12, 4] for details) can be applied to the continuous scores after training.”
The fused lasso penalty for learning interpretable medical scoring systems (2017, methodology): “They made a first attempt to implement online learning with ranomized rounding, where the main motivation was to reduce the memory needed to store the vector of coefficients of a parametric model, since storing all the coefficients consumes considerable RAM [10].”