Review of Input Variable Selection Methods for Artificial Neural Networks

By Robert J. May, Graeme C. Dandy, Holger R. Maier. 2011

389 citations recorded by the source corpus.

The choice of input variables is a fundamental, and yet crucial consideration in identifying the optimal functional form of statistical models. The task of selecting input variables is common to the development of all statistical models, and is largely dependent on the discovery of relationships within the available data to identify suitable predictors of the model output. In the case of parametric, or semi-parametric empirical models, the difficulty of the input variable selection task is somewhat alleviated by the a priori assumption of the functional form of the model, which is based on some physical interpretation of the underlying system or process being modelled. However, in the case of artificial neural networks (ANNs), and other similarly data-driven statistical modelling approaches, there is no such assumption made regarding the structure of the model. Instead, the input variables are selected from the available data, and the model is developed subsequently. The difficulty of selecting input variables arises due to (i) the number of available variables, which may be very large; (ii) correlations between potential input variables, which creates redundancy; and (iii) variables that have little or no predictive power. Variable subset selection has been a longstanding issue in fields of applied statistics dealing with inference and linear regression (Miller, 1984), and the advent of ANN models has only served to create new challenges in this field. The non-linearity, inherent complexity and non-parametric nature of ANN regression make it difficult to apply many existing analytical variable selection methods. The difficulty of selecting input variables is further exacerbated during ANN development, since the task of selecting inputs is often delegated to the ANN during the learning phase of development. A popular notion is that an ANN is adequately capable of identifying redundant and noise variables during training, and that the trained network will use only the salient input variables. ANN architectures can be built with arbitrary flexibility and can be successfully trained using any combination of input variables (assuming they are good predictors). Consequently, allowances are often made for a large number of input variables, with the belief that the ability to incorporate such flexibility and redundancy creates a more robust model. Such pragmatism is perhaps symptomatic of the popularisation of ANN models through machine learning, rather than statistical learning theory. ANN models are too often developed without due consideration given to the effect that the choice of input variables has on model complexity, learning difficulty, and performance of the subsequently trained ANN. 1

Subfields: Artificial Intelligence, Control and Systems Engineering, Environmental Engineering

View Paper: https://www.semanticscholar.org/paper/977f9694f10f25a499e2d32f660da85e4ca183de

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Control and Systems Engineering

In Environmental Engineering

Citations in the corpus

Bootstrap rank‐ordered conditional mutual information (broCMI): A nonlinear input variable selection method for water resources modeling (2016, methodology): “Filters are the most commonly used input variable selection method since they may be applied independent of the regression model (e.g., MLR or ANN) while both wrappers and embedded methods rely on a model in order to choose the final set of input variables [May et al., 2011].”
Review of soft sensor methods for regression applications (2016, background): “ded and wrapper approaches in some cases [67].”
Secchi Disk Depth Estimation from Water Quality Parameters: Artificial Neural Network versus Multiple Linear Regression Models? (2016, methodology): “Contrary to empirical models, in which the physical interpretations of the problem in question can help considerably in the selection of the relevant input and are often influential in the determination of the final topological structure of the models, this it is not the case in ANN models: prior kn”
Using deep neural network with small dataset to predict material defects (2019, background): “Principal component analysis (PCA) is known to reduce data dimensions and noise in the input data [71].”
Fast relational learning using bottom clause propositionalization with artificial neural networks (2013, methodology): “Alternatively, statistical methods such as Pearson’s correlation and Principal Component Analysis can be used (a survey of those methods can be found in May et al. (2011)), taking advantage of the use of numerical feature vectors as training patterns.”