Hopular: Modern Hopfield Networks for Tabular Data

By Bernhard Schäfl, Lukas Gruber, Angela Bitto-Nemling, Sepp Hochreiter. 2022

33 citations recorded by the source corpus.

While Deep Learning excels in structured data as encountered in vision and natural language processing, it failed to meet its expectations on tabular data. For tabular data, Support Vector Machines (SVMs), Random Forests, and Gradient Boosting are the best performing techniques with Gradient Boosting in the lead. Recently, we saw a surge of Deep Learning methods that were tailored to tabular data but still underperform compared to Gradient Boosting on small-sized datasets. We suggest"Hopular", a novel Deep Learning architecture for medium- and small-sized datasets, where each layer is equipped with continuous modern Hopfield networks. The modern Hopfield networks use stored data to identify feature-feature, feature-target, and sample-sample dependencies. Hopular's novelty is that every layer can directly access the original input as well as the whole training set via stored data in the Hopfield networks. Therefore, Hopular can step-wise update its current model and the resulting prediction at every layer like standard iterative learning algorithms. In experiments on small-sized tabular datasets with less than 1,000 samples, Hopular surpasses Gradient Boosting, Random Forests, SVMs, and in particular several Deep Learning methods. In experiments on medium-sized tabular data with about 10,000 samples, Hopular outperforms XGBoost, CatBoost, LightGBM and a state-of-the art Deep Learning method designed for tabular data. Thus, Hopular is a strong alternative to these methods on tabular data.

Subfields: Computer Vision and Pattern Recognition, Media Technology, Model Architectures, Recommender Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

In Model Architectures

Citations in the corpus

History Compression via Language Models in Reinforcement Learning (2022, background): “FrozenHopfield is an instance of a MHN (Ramsauer et al., 2021; Fürst et al., 2021; Schäfl et al., 2022) that can associate image observations (or any other input) with pretrained token embeddings (Mikolov et al.”
When Do Neural Nets Outperform Boosted Trees on Tabular Data? (2023, methodology): “The regression datasets used in these experiments have been used in recent studies of machine learning with tabular data [26, 34, 47, 53]; each dataset corresponds to an OpenML task, and can be preprocessed exactly like the classification datasets used in other experiments.”