Survey on categorical data for neural networks

By John T. Hancock, Taghi M. Khoshgoftaar. Journal of Big Data, 2020

542 citations recorded by the source corpus.

This survey investigates current techniques for representing qualitative data for use as input to neural networks. Techniques for using qualitative data in neural networks are well known. However, researchers continue to discover new variations or entirely new methods for working with categorical data in neural networks. Our primary contribution is to cover these representation techniques in a single work. Practitioners working with big data often have a need to encode categorical values in their datasets in order to leverage machine learning algorithms. Moreover, the size of data sets we consider as big data may cause one to reject some encoding techniques as impractical, due to their running time complexity. Neural networks take vectors of real numbers as inputs. One must use a technique to map qualitative values to numerical values before using them as input to a neural network. These techniques are known as embeddings, encodings, representations, or distributed representations. Another contribution this work makes is to provide references for the source code of various techniques, where we are able to verify the authenticity of the source code. We cover recent research in several domains where researchers use categorical data in neural networks. Some of these domains are natural language processing, fraud detection, and clinical document automation. This study provides a starting point for research in determining which techniques for preparing qualitative data for use with neural networks are best. It is our intention that the reader should use these implementations as a starting point to design experiments to evaluate various techniques for working with qualitative data in neural networks. The third contribution we make in this work is a new perspective on techniques for using categorical data in neural networks. We organize techniques for using categorical data in neural networks into three categories. We find three distinct patterns in techniques that identify a technique as determined, algorithmic, or automated. The fourth contribution we make is to identify several opportunities for future research. The form of the data that one uses as an input to a neural network is crucial for using neural networks effectively. This work is a tool for researchers to find the most effective technique for working with categorical data in neural networks, in big data settings. To the best of our knowledge this is the first in-depth look at techniques for working with categorical data in neural networks.

Subfields: Artificial Intelligence

View Paper: https://www.semanticscholar.org/paper/940874fb53709e54918ea91f6d00112fbed5f01b

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

Citations in the corpus

Quantum Computing for Finance: State-of-the-Art and Future Prospects (2020, background): “There have been many proposed encodings, with one-hot encoding as one of the most populars, for such purposes [109].”
Deep Neural Networks and Tabular Data: A Survey (2021, methodology): “There are many different possibilities to transform tabular data, and each may have a different impact on the learning results [47].”
Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “…been various attempts to make deep learning work on tabular data: data encoding techniques to make tabular data better suited for deep learning [Hancock and Khoshgoftaar, 2020, Yoon et al., 2020], "hybrid methods" to beneﬁt from the ﬂexibility of NNs while keeping the inductive biases of other…”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, unknown): “…have been various attempts to adapt deep learning to tabular data: data encoding techniques to make tabular data better suited for deep learning [Hancock and Khoshgoftaar, 2020, Yoon et al., 2020], "hybrid methods" to benefit from the flexibility of neural networks while keeping the inductive…”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “Data transformation methods [29, 64] seek to encode the data into a format that is better-suited for neural nets.”