Learning Feature Engineering for Classification

By Fatemeh Nargesian, Horst Samulowitz, Udayan Khurana, Elias Boutros Khalil, Deepak S. Turaga. 2017

300 citations recorded by the source corpus.

Feature engineering is the task of improving predictive modelling performance on a dataset by transforming its feature space. Existing approaches to automate this process rely on either transformed feature space exploration through evaluation-guided search, or explicit expansion of datasets with all transformed features followed by feature selection. Such approaches incur high computational costs in runtime and/or memory. We present a novel technique, called Learning Feature Engineering (LFE), for automating feature engineering in classification tasks. LFE is based on learning the effectiveness of applying a transformation (e.g., arithmetic or aggregate operators) on numerical features, from past feature engineering experiences. Given a new dataset, LFE recommends a set of useful transformations to be applied on features without relying on model evaluation or explicit feature expansion and selection. Using a collection of datasets, we train a set of neural networks, which aim at predicting the transformation that impacts classification performance positively. Our empirical results show that LFE outperforms other feature engineering approaches for an overwhelming majority (89%) of the datasets from various sources while incurring a substantially lower computational cost.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Marco F. Huber
2. Marc-André Zöller
3. Quanming Yao
4. Xia Hu
5. Qingquan Song
6. Wei-Wei Tu
7. Yi-Wei Chen
8. Yang Yu

Citations in the corpus

• Benchmark and Survey of Automated Machine Learning Frameworks (2019, methodology): “Based on the metafeatures of a candidate feature the expected loss reduction after including this candidate can be predicted using a regression model (Katz et al., 2017; Nargesian et al., 2017), reinforcement learning (Khurana et al.”
• Survey on Automated Machine Learning (2019, methodology): “Based on the meta-features of a candidate feature, but also meta-features of its parents, the expected loss reduction after including this candidate can be predicted using a regression model (Katz et al., 2017; Nargesian et al., 2017).”
• Automated Machine Learning in Practice: State of the Art and Recent Results (2019, methodology): “To reduce the computational complexity of iterative feature selection methods, Learning Feature Engineering (LFE) [20] learns the effectiveness of a transformation based on previous experiments.”
• Techniques for Automated Machine Learning (2019, methodology): “Edges are SIGKDD Explorations Volume 22, Issue 2 37 Category Technique Bayesian Optimization Reinforcement Learning Evolutionary Algorithm Gradient Approach Otherss Automated Feature Engineering (AutoFE) - FeatureRL [49] NFS [14] Genetic Programming Feature Engineering [84] Genetic Programming Skewe”
• Human-AI Collaboration in Data Science (2019, background): “There are also techniques that try to engineer new features for a target ML model using reinforcement learning [37] and guided search based on past experiences [49].”
• Automatic Model Monitoring for Data Streams (2019, background): “ods to stages such as data cleaning and pre-processing [31], feature engineering and selection [16, 26], and more recently, model monitoring [22].”
• Taking Human out of Learning Applications: A Survey on Automated Machine Learning (2018, background): “feature engineering subsequent classifiers feature sets [9], [10], [45], [46], [47] selection of methods and their hyper-parameters [5], [6], [34]”
• A Survey on Deep Learning for Data-Driven Soft Sensors (2021, background): “However, the forms of features are still handcrafted based on knowledge and experience, which is called as feature engineering [25], [26].”