Tune: A Research Platform for Distributed Model Selection and Training
By Richard Liaw, Eric Liang, Robert Nishihara, Philipp Moritz, Joseph E. Gonzalez, Ion Stoica. 2018
1,129 citations recorded by the source corpus.
Modern machine learning algorithms are increasingly computationally demanding, requiring specialized hardware and distributed computation to achieve high performance in a reasonable time frame. Many hyperparameter search algorithms have been proposed for improving the efficiency of model selection, however their adaptation to the distributed compute environment is often ad-hoc. We propose Tune, a unified framework for model selection and training that provides a narrow-waist interface between training scripts and search algorithms. We show that this interface meets the requirements for a broad range of hyperparameter search algorithms, allows straightforward scaling of search to large clusters, and simplifies algorithm implementation. We demonstrate the implementation of several state-of-the-art hyperparameter search algorithms in Tune. Tune is available at this http URL
Subfields: Artificial Intelligence, Neural Architecture Search & AutoML, ML Systems & Hardware
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Quoc V. Le
- Yoshua Bengio
- Percy Liang
- Mohit Bansal
- Noah A. Smith
- Yejin Choi
- Jianfeng Gao
- Kyunghyun Cho
In Neural Architecture Search & AutoML
In ML Systems & Hardware
- Gaurav Menghani
- Sanjay Krishnan
- Eugene Wu
- Pratik Chaudhari
- Noah A. Smith
- Ameet Talwalkar
- Premkumar Natarajan
- Jia Deng
Citations in the corpus
- FBNetV3: Joint Architecture-Recipe Search using Neural Acquisition Function (2020, background): “Several deep-learning-specific tuning libraries are also widely used [20].”
- Hyper-Parameter Optimization: A Review of Algorithms and Applications (2020, methodology): “Median stopping is the most straightforward early termination policy, adapted by structured HPO frameworks such as Google Vizier (Golovin et al., 2017), Tune (Liaw et al., 2018), and NNI (Microsoft, 2018).”
- AlphaClean: Automatic Generation of Data Cleaning Pipelines (2019, methodology): “Since Bayesian optimization is inherently sequential, for parallel and distributed settings, the community is increasingly studying randomized and grid search schemes [20, 32, 33].”
- Model Zoo: A Growing Brain That Learns Continually (2021, methodology): “Ray Tune (Liaw et al., 2018) was used for hyper-parameter tuning using a multi-task learning model on all tasks from Coarse CIFAR-100.”
- A System for Massively Parallel Hyperparameter Tuning (2018, methodology): “These existing systems provide generic support for a wide range of hyperpa-rameter tuning algorithms; both RayTune and Optuna in fact have support for ASHA.”
- Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Better (2021, methodology): “NNI [131], Tune [103], Advisor [31] are other open-source HPO software packages that can be used locally.”
- MASSIVE: A 1M-Example Multilingual Natural Language Understanding Dataset with 51 Typologically-Diverse Languages (2022, methodology): “…(ASHA) (Li et al., 2018a) for scheduling, which are both part of the hyperopt library (Bergstra et al., 2013) integrated into the ray[tune] library (Liaw et al., 2018), which is itself integrated into the Trainer from the transformers library (Wolf et al., 2020), which we used for modeling and for…”
- GRAND: Graph Neural Diffusion (2021, methodology): “Hyperparameter search used Ray Tune (Liaw et al., 2018) with a thousand random trials using an asynchronous hyperband scheduler with a grace period of ten epochs and a half life of ten epochs.”
- PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization (2023, background): “Critical factors such as optimizer selection, learning rate, number of training epochs, and quality and size of training data significantly influence the model’s performance [21, 22].”
- Health system-scale language models are all-purpose prediction engines (2023, methodology): “The hyperpa-rameters and evaluation and software libraries for fine-tuning NYUTron variants were the same as for fine-tuning NYUTron.”
- Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance (2023, background): “Since this trend towards increasingly complex experimentation adds more and more parameters to tune, many parameter-tuning frameworks have been developed so far such as Optuna (Akiba et al., 2019), Ray (Liaw et al., 2018), BoTorch (Balandat et al., 2020), and Hyperopt (Bergstra et al., 2011, 2013a, ”
- Variable Bitrate Discrete Neural Representations via Causal Self-Attention (2021, methodology): “We will optimize each model using LAMB [9], performing hyperparameter search over training hyperparameters (learning rate, scheduling, warmup, batch size, epochs, and regularization) via Tune [13] and selecting based on best reconstruction quality on a held-out dev set.”
- Optuna: A Next-generation Hyperparameter Optimization Framework (2019, methodology): “ecient pruning strategy, when in fact both parameter searching strategy and performance estimation strategy are important for high-performance optimization under limited resource availability [12][5][7]. Finally, in order to accommodate with a variety of models in a variety of situations, the archit”
- Techniques for Automated Machine Learning (2019, background): “With the integration of friendly user interfaces [57], AutoML users could also customize their solutions by selecting desirable configurations between independent runs.”
- Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Prediction (2021, methodology): “We use ray.tune [Liaw et al., 2018] for hyperparameter optimization and track experiments with W&B [Biewald, 2020].”