Entity Embeddings of Categorical Variables

By Cheng Guo, Felix Berkhahn. 2016

457 citations recorded by the source corpus.

We map categorical variables in a function approximation problem into Euclidean spaces, which are the entity embeddings of the categorical variables. The mapping is learned by a neural network during the standard supervised training process. Entity embedding not only reduces memory usage and speeds up neural networks compared with one-hot encoding, but more importantly by mapping similar values close to each other in the embedding space it reveals the intrinsic properties of the categorical variables. We applied it successfully in a recent Kaggle competition and were able to reach the third position with relative simple features. We further demonstrate in this paper that entity embedding helps the neural network to generalize better when the data is sparse and statistics is unknown. Thus it is especially useful for datasets with lots of high cardinality features, where other methods tend to overfit. We also demonstrate that the embeddings obtained from the trained neural network boost the performance of all tested machine learning methods considerably when used as the input features instead. As entity embedding defines a distance measure for categorical variables it can be used for visualizing categorical data and for data clustering.

Subfields: Artificial Intelligence

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

1. Alex Smola
2. Mu Li
3. Pratik Chaudhari
4. Xiangnan He
5. Rasool Fakoor
6. Yong Li
7. Chen Gao
8. Alan L. Yuille

Citations in the corpus

• An overview of deep learning in medical imaging focusing on MRI (2018, background): “They are, however, not limited to image processing and analysis but are outperforming other approaches in areas like natural language processing [3, 4, 5], speech recognition and synthesis [6, 7]2, and in the analysis of unstructured, tabular-type data using entity embeddings [8, 9].”
• AutoGluon-Tabular: Robust and Accurate AutoML for Structured Data (2020, background): “Our network applies a separate embedding layer to each categorical feature (Guo and Berkhahn, 2016) which enables this model to separately learn about each categorical feature before its representation is blended with other variables.”
• Fast, Accurate, and Simple Models for Tabular Data via Augmented Distillation (2020, background): “This network uses a separate embedding layer for each categorical feature which helps the network separately learn about each of these variables before their representations are blended together by fully-connected layers [59, 63].”
• Sequence classification for credit-card fraud detection (2018, methodology): “[2014] and is applicable to arbitrary other categorical variables apart from words [Guo and Berkhahn, 2016].”
• Deep Neural Networks and Tabular Data: A Survey (2021, unknown): “A similar work by Guo and Berkhahn [99] proposes an embedding using deep neural networks for categorical variables.”
• Survey on categorical data for neural networks (2020, unknown): “Entity embedding is a method to embed categorical variables in real-valued vector spaces [3].”
• Time-to-Event Prediction with Neural Networks and Cox Regression (2019, methodology): “For the neural networks, we standardize the numerical covariates, and encode the categorical covariates by entity embeddings (Guo and Berkhahn, 2016) half the size of the number of categories.”
• CLIP-Driven Universal Model for Organ Segmentation and Tumor Detection (2023, background): “[19], one- or few-hot labels are replaced by the text embedding generated by the pre-trained text encoder from CLIP1.”
• λOpt: Learn to Regularize Recommender Models in Finer Levels (2019, methodology): “Embedding technique is widely used to project categorical values into a latent vector space [13].”