Optuna: A Next-generation Hyperparameter Optimization Framework
By Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, Masanori Koyama. 2019
10,668 citations recorded by the source corpus.
The purpose of this study is to introduce new design-criteria for next-generation hyperparameter optimization software. The criteria we propose include (1) define-by-run API that allows users to construct the parameter search space dynamically, (2) efficient implementation of both searching and pruning strategies, and (3) easy-to-setup, versatile architecture that can be deployed for various purposes, ranging from scalable distributed computing to light-weight experiment conducted via interactive interface. In order to prove our point, we will introduce Optuna, an optimization software which is a culmination of our effort in the development of a next generation optimization software. As an optimization software designed with define-by-run principle, Optuna is particularly the first of its kind. We will present the design-techniques that became necessary in the development of the software that meets the above criteria, and demonstrate the power of our new design through experimental results and real world applications. Our software is available under the MIT license (https://github.com/pfnet/optuna/).
Subfields: Artificial Intelligence, Computational Theory and Mathematics, Neural Architecture Search & AutoML
How researchers see this paper
The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.
In Artificial Intelligence
- Jianfeng Gao
- Mohit Bansal
- Percy Liang
- Yoshua Bengio
- Luke S. Zettlemoyer
- Quoc V. Le
- Yejin Choi
- Richard Socher
In Computational Theory and Mathematics
- Anne-Laure Boulesteix
- Shuhei Watanabe
- Shion Honda
- Tobias Pielok
- Jascha Narain Sohl-Dickstein
- Qian Liu
- Shuicheng Yan
- Artem Babenko
In Neural Architecture Search & AutoML
- Colin White
- Tsubasa Hirakawa
- Tobias Pielok
- Shuicheng Yan
- Artem Babenko
- Martin Binder
- Michel Lang
- Jakob Richter
Citations in the corpus
- Learning Visual Representations with Caption Annotations (2020, neutral): “where patt ∈ [0, 1]H·W such that patt1H·W = 1.”
- Isometric Transformation Invariant and Equivariant Graph Convolutional Networks (2020, methodology): “We applied the Adam optimizer (Kingma & Ba, 2014) and Optuna (Akiba et al., 2019) to optimize the learning rate.”
- Towards Robust and Reproducible Active Learning using Neural Networks (2020, methodology): “We make use of the Optuna library [1] to facilitate these experiments.”
- Revisiting Deep Learning Models for Tabular Data (2021, methodology): “For most algorithms, we use the Optuna library (Akiba et al., 2019) to run Bayesian optimization (the Tree-Structured Parzen Estimator algorithm), which is reported to be superior to random search (Turner et al., 2021).”
- Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methods (2021, methodology): “[2] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.”
- Evolution of Heuristics: Towards Efficient Automatic Algorithm Design Using Large Language Model (2024, neutral): “With the development of various effective methodoliges (Blot et al., 2016; L´opez-Ib´a˜nez et al., 2016; Akiba et al., 2019) and frameworks (Burke et al., 2019), one can tune heuristics or combine different algorithmic components in an automatic manner.”
- Hyperparameter optimization: Foundations, algorithms, best practices, and open challenges (2021, background): “One of the most popular and well-maintained HPO tools is Optuna (Akiba et al., 2019).”
- Practical tradeoffs between memory, compute, and performance in learned optimizers (2022, background): “, 2012), or other low-dimensional black-box optimization techniques (Bergstra et al., 2011; Golovin et al., 2017; Akiba et al., 2019).”
- DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines (2023, methodology): “…using model selection techniques like cross-validation or, in principle, with sophisticated techniques involving RL and LM feedback (Hu et al., 2023; Zhao et al., 2023a; Shinn et al., 2023) or learned or Bayesian hyperparameter optimization methods (Bergstra et al., 2013; Akiba et al., 2019).”
- COMET-22: Unbabel-IST 2022 Submission for the Metrics Shared Task (2022, methodology): “Our primary submission is an ensemble between a C OMET Estimator model trained on top of XLM-R using DA from 2017 to 2020 and a sequence tagging model, such as the one described above, trained on top of InfoXLM (Chi et al., 2021).”
- Data Augmentation for Graph Neural Networks (2020, methodology): “We report test microF1 scores over 30 runs, employing Optuna (Akiba et al. 2019) for efficient hyperparameter search.”
- Deep Neural Networks and Tabular Data: A Survey (2021, methodology): “1) Hyperparameter Selection: In order to do a fair evaluation, we use the Optuna library [203] with 100 iterations”
- When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “During this time, we train and evaluate the algorithm with at most 30 hyperparameter sets (one default set and 29 random sets, using Optuna [2]).”
- Predicting locations of cryptic pockets from single protein structures using the PocketMiner graph neural network (2023, methodology): “We tuned these hyperparameters using the optuna software package 51 but found little sensitivity to the choice of these hyperparameters (Supplementary Table 2, see “ Model Training – Task 1 ” for detailed information about hyperparameter selection).”
- SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery (2019, methodology): “We choose the model and hyperpa-rameter set achieving the best validation score with optuna (Akiba et al. 2019), from a linear model with L2 penalty, et al. 2017).”