On Embeddings for Numerical Features in Tabular Deep Learning

By Yura Gorishniy, Ivan Rubachev, Artem Babenko. Neural Information Processing Systems, 2022

308 citations recorded by the source corpus.

Recently, Transformer-like deep architectures have shown strong performance on tabular data problems. Unlike traditional models, e.g., MLP, these architectures map scalar values of numerical features to high-dimensional embeddings before mixing them in the main backbone. In this work, we argue that embeddings for numerical features are an underexplored degree of freedom in tabular DL, which allows constructing more powerful DL models and competing with GBDT on some traditionally GBDT-friendly benchmarks. We start by describing two conceptually different approaches to building embedding modules: the first one is based on a piecewise linear encoding of scalar values, and the second one utilizes periodic activations. Then, we empirically demonstrate that these two approaches can lead to significant performance boosts compared to the embeddings based on conventional blocks such as linear layers and ReLU activations. Importantly, we also show that embedding numerical features is beneficial for many backbones, not only for Transformers. Specifically, after proper embeddings, simple MLP-like models can perform on par with the attention-based architectures. Overall, we highlight embeddings for numerical features as an important design aspect with good potential for further improvements in tabular DL.

View Paper: https://www.semanticscholar.org/paper/1057c047156594c8494b7e5f1f747d9f15dd1162

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

Citations in the corpus

Why do tree-based models still outperform deep learning on tabular data? (2022, unknown): “Our ﬁndings shed light on the results of Somepalli et al. [2021] and Gorishniy et al. [2022], which add an embedding layer, even for numerical features, before MLP or Transformer models.”
TabLLM: Few-shot Classification of Tabular Data with Large Language Models (2022, unknown): “Other improvements could include more faithful LLM serializations as well as numeric-speciﬁc encoding methods (Gorishniy et al., 2022).”
Why do tree-based models still outperform deep learning on typical tabular data? (2022, unknown): “…observation could also explain the benefits of the ExU activation used in the Neural-GAM paper [Agarwal et al., 2021], and of the embeddings used in Gorishniy et al. [2022]: the periodic embedding might help the model to learn the high-frequency part of the target function, and the target-aware…”
Deep Neural Networks and Tabular Data: A Survey (2021, background): “However, for tabular data and deep neural networks, the performance may strongly depend on the selected preprocessing strategy [75].”
Griffin: Towards a Graph-Centric Relational Database Foundation Model (2025, unknown): “Other works have explored improved encoding strategies for numerical features (Gorishniy et al., 2022; Yarullin & Isaev, 2023), while some have highlighted the benefits of incorporating nearest-neighbor information (Gorishniy et al., 2023; Ye et al., 2025).”