Probabilistic Matrix Factorization for Automated Machine Learning

By Nicoló Fusi, Rishit Sheth, Melih Elibol. 2017

146 citations recorded by the source corpus.

In order to achieve state-of-the-art performance, modern machine learning techniques require careful data pre-processing and hyperparameter tuning. Moreover, given the ever increasing number of machine learning models being developed, model selection is becoming increasingly important. Automating the selection and tuning of machine learning pipelines consisting of data pre-processing methods and machine learning models, has long been one of the goals of the machine learning community. In this paper, we tackle this meta-learning task by combining ideas from collaborative filtering and Bayesian optimization. Using probabilistic matrix factorization techniques and acquisition functions from Bayesian optimization, we exploit experiments performed in hundreds of different datasets to guide the exploration of the space of possible pipelines. In our experiments, we show that our approach quickly identifies high-performing pipelines across a wide range of datasets, significantly outperforming the current state-of-the-art.

Subfields: Artificial Intelligence, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

Citations in the corpus

Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “AutoML Azure uses collaborative filtering and Bayesian optimization to search for the most promising pipelines efficiently [43] based on a database that is constructed by running millions of experiments of evaluation of different pipelines on many datasets.”
Meta-Learning: A Survey (2018, positive): “3 can be seen as using a distance-based meta-model to warm-start Bayesian optimization [48, 54] or evolutionary algorithms [59, 129].”
Towards human-guided machine learning (2019, methodology): “The approach was adopted for the scikit-learn library [8] in Auto-sklearn [2], extended with meta-learning and ensemble construction, and later with probabilistic matrix factorization [4].”
Bayesian tuning and bandits : an extensible, open source library for AutoML (2018, methodology): “In [14], the authors use probabilistic matrix factorization to infer the data missing from the matrix (including the new dataset for which they wish to propose a pipeline).”
Auto-Sklearn 2.0: Hands-free AutoML via Meta-Learning (2020, methodology): “Extending these portfolio strategies which are learned offline, there are online portfolios which can select from a fixed set of machine learning pipelines, taking previous evaluations into account (Leite et al., 2013; Wistuba et al., 2015a,b; Fusi et al., 2018; Yang et al., 2019, 2020).”
Neural Architecture Search with Reinforce and Masked Attention Autoregressive Density Estimators (2020, background): “A limitation of these approaches is that performance degrades in high-dimensional search spaces, because larger samples are required to update the posterior distribution (Fusi et al., 2018).”
D-VAE: A Variational Autoencoder for Directed Acyclic Graphs (2019, methodology): “Fusi et al. [37] proposed a novel AutoML algorithm also using model embedding, but with a matrix factorization approach.”
Fast Task-Aware Architecture Inference (2019, background): “In general, previous works can be broadly categorized into the following classes: • Bayesian optimization methods [19, 14, 9, 6] build a probabilistic model of the performance of the network as a function of its hyperparameters and then decide which candidate point in the search space to evaluate…”
Techniques for Automated Machine Learning (2019, methodology): “Furthermore, Azure AutoML uses collaborative filtering and Bayesian optimization to search for promising pipelines [28].”
Learning search spaces for Bayesian optimization: Another view of hyperparameter transfer learning (2019, methodology): “The proposed approach relies on the observation that HPO problems occurring in ML are often related (for example, tuning the HPs of an ML model trained on different data sets [43, 3, 51, 35, 11, 32, 9, 28]).”
FLAML: A Fast and Lightweight AutoML Library (2019, background): “…assumption that one can collect a large number of datasets and experiments for meta-training, and the performance of learners and hyperparameters from these experiments is indicative of their future performance in new datasets and tasks (Feurer et al., 2015; Fusi et al., 2018; Shang et al., 2019).”
P4ML: A Phased Performance-Based Pipeline Planner for Automated Machine Learning (2018, methodology): “While Feurer et al. (2015a) use random forests as a surrogate, recent advances have been achieved by using matrix factorization (Fusi et al., 2017) or scaling Gaussian process surrogates to large collections of metadata (Wistuba et al., 2018).”
Multiple Adaptive Bayesian Linear Regression for Scalable Bayesian Optimization with Warm Start (2017, background): “…work, we are interested in such a setting as we would like to warm start BO by, e.g., transferring information obtained from previous runs of the BO routine, or learn across similar problems (e.g., a given classiﬁer applied across different datasets [2, 25, 8, 9, 10]), which we will call tasks .”
Automatic Machine Learning by Pipeline Synthesis using Model-Based Reinforcement Learning and a Grammar (2019, background): “…et al., 2017), Bayesian optimization (Rasmussen, 2003; Bergstra and Bengio, 2012; Feurer et al., 2015; Kotthoﬀ et al., 2017), collaborative ﬁltering (Yang et al., 2018; Fusi et al., 2018), and grammars (de S´a et al., 2017), for ﬁnding machine learning pipelines for a given dataset and task.”
Democratizing Data Science through Interactive Curation of ML Pipelines (2019, methodology): “Most industry cloud ML services, such as TensorFlow [2], Amazon SageMaker and Azure Machine Learning [27], fall into this category, in that they provide fully-managed environment for ML applications.”
pTNAS: Progressive Neural Architecture Search for Tabular Data (2024, unknown): “(Levin et al., 2023; Gorishniy et al., 2021; Luo et al., 2021; Zhu et al., 2026), Automated Machine Learning (AutoML) on tabular data (Fusi et al., 2018; Olson & Moore, 2016; Yang et al., 2019), to Tabular Foundation Models (Qu et al., 2025; Hollmann et al., 2023; Robertson et al., 2025).”