Ranking Learning Algorithms: Using IBL and Meta-Learning on Accuracy and Time Results

By Pavel Brazdil, Carlos Soares, Joaquim F. Pinto da Costa. 2003

473 citations recorded by the source corpus.

Subfields: Artificial Intelligence, Information Systems, Statistics and Probability, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Statistics and Probability

In Evaluation & Benchmarks

Citations in the corpus

Learning to Optimize Neural Nets (2017, background): “Other meth-ods (Brazdil et al., 2003; Schmidhuber, 2004) aim to learn how to choose a base-level learner that would perform the best on a given task.”
Multi-Fidelity Automatic Hyper-Parameter Tuning via Transfer Series Expansion (2019, background): “Model conﬁguration is often considered as model selection (Brazdil, Soares, and Da Costa 2003; Chapelle, Vap-nik, and Bengio 2002; Maron and Moore 1994; Zhao and Yu 2006) and hyper-parameter tunning (Bergstra and Ben-gio 2012; Snoek, Larochelle, and Adams 2012).”
Automated Machine Learning: State-of-The-Art and Open Challenges (2019, methodology): “Meta models can also be used to rank a particular set of configurations by using the K−nearest neighbour model on the meta features of prior tasks and predicting the top k tasks that are similar to new task tnew and then ranking the best set of configurations of these similar tasks [19, 32].”
One-shot Learning with Memory-Augmented Neural Networks (2016, methodology): “Although the term has been used in numerous senses (Schmidhuber et al., 1997; Caruana, 1997; Schweighofer & Doya, 2003; Brazdil et al., 2003), meta-learning generally refers to a scenario in which an agent learns at two levels, each associated with different time scales.”
Meta-Learning: A Survey (2018, neutral): “To find the best θ∗ for a task tnew, never before seen, a simple anytime method is to select the top-K configurations [21], going down the list and evaluating each configuration on tnew in turn.”
Supervised Machine Learning: A Review of Classification Techniques (2007, unknown): “Some characteristics of learning tasks are: the number of instances, the proportion of categorical attributes, the proportion of missing values, the entropy of classes, etc. Brazdil et al. (2003) provided an extensive list of information and statistical measures for a dataset.”
Predicting the Performance of Learning Algorithms Using Support Vector Machines as Meta-regressors (2008, background): “Although the strict Meta-Learning approach (as described above) has been applied by different authors (such as [2, 4, 9, 14, 15]), certain information loss may be introduced in the definition of the class labels associated to meta-examples.”
Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, background): “The meta-learner is trained to map the meta-features to the performance [50], [99], [100], applicability [101], [102], or ranking [103], [104], [105], [106] of models.”
Meta Filter Pruning to Accelerate Deep Convolutional Neural Networks (2019, methodology): “The general, statistical and information theoretic measures to characterize the datasets are called meta-attributes of the the datasets [1].”
Meta-Learning with Memory-Augmented Neural Networks (2016, methodology): “Although the term has been used in numerous senses (Schmidhuber et al., 1997; Caruana, 1997; Schweighofer & Doya, 2003; Brazdil et al., 2003), meta-learning generally refers to a scenario in which an agent learns at two levels, each associated with different time scales.”
Metalearning: a survey of trends and technologies (2013, methodology): “One of the simplest ranking method involves dividing the problem space using clustering of available datasets according to a distance measure (usually k -Nearest Neighbour) of the metafeatures and using average performance ranks of the cluster into which a new problem falls [11].”
Label ranking by learning pairwise preferences (2008, methodology): “Yet another application scenario is meta-learning, where the task is to rank learning algorithms according to their suitability for a new dataset, based on the characteristics of this dataset (Brazdil et al., 2003).”
Selection of time series forecasting models based on performance information (2004, methodology): “In this work, the meta-attributes used were the length of the series, basic tendency, percentage of turning points and first auto-correlation coefficient.”
Using Meta-Learning to Support Data Mining (2004, methodology): “Different approaches exist to evaluate methods that predict rankings of learning algorithms [12].”
Automatic classifier selection for non-experts (2014, methodology): “[8] used a k-Nearest Neighbor approach for creating the ranking.”
Auto-WEKA: combined selection and hyperparameter optimization of classification algorithms (2012, background): “, and becomes quickly impractical as the number of candidate algorithms grows. One alternative solution is to provide the user with a ranking of suggested algorithms based on data set characteristics [8]. However, this approach does not avert the need to optimize hyper-parameters and to select among”