Learning to Embed Categorical Features without Embedding Tables for Recommendation

By Wang-Cheng Kang, Derek Zhiyuan Cheng, Tiansheng Yao, Xinyang Yi, Ting-Li Chen, Lichan Hong, Ed H. Chi. 2020

88 citations recorded by the source corpus.

Embedding learning of categorical features (e.g. user/item IDs) is at the core of various recommendation models. The standard approach creates an embedding table where each row represents a dedicated embedding vector for every unique feature value. However, this method fails to efficiently handle high-cardinality features and unseen feature values (e.g. new video ID) that are prevalent in real-world recommendation systems. In this paper, we propose an alternative embedding framework Deep Hash Embedding (DHE), replacing embedding tables by a deep embedding network to compute embeddings on the fly. DHE first encodes the feature value to a unique identifier vector with multiple hashing functions and transformations, and then applies a DNN to convert the identifier vector to an embedding. The encoding module is deterministic, non-learnable, and free of storage, while the embedding network is updated during the training time to learn embedding generation. Empirical results show that DHE achieves comparable AUC against the standard one-hot full embedding, with smaller model sizes. Our work sheds light on the design of DNN-based alternative embedding schemes for categorical features without using embedding table lookup.

Subfields: Computer Vision and Pattern Recognition, Information Systems, Representation & Self-Supervised Learning

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Computer Vision and Pattern Recognition

1. Yi Tay
2. Carole-Jean Wu
3. Michael G. Rabbat
4. Vinh Q. Tran
5. Ed H. Chi
6. Lichan Hong
7. Shashank Rajput
8. Christos Kozyrakis

In Information Systems

1. Hongzhi Yin
2. Tong Chen
3. Xin Xia
4. Junliang Yu
5. Zi-Liang Huang
6. Carole-Jean Wu
7. Bilge Acun
8. Ed H. Chi

In Representation & Self-Supervised Learning

1. Shashank Rajput
2. Yi Tay
3. Zi-Liang Huang
4. Tong Chen
5. Michael G. Rabbat
6. Jundong Li
7. Nikhil Mehta
8. Bilge Acun

Citations in the corpus

• Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “Similarly, the design space trade-off between memory capacity requirement, training time, and model accuracy is also explored in Deep Hash Embedding (DHE) [57].”
• Self-Supervised Learning for Recommender Systems: A Survey (2022, background): “Meanwhile, the problems like the highly-skewed data distribution [38], widely observed biases [39], and large-vocabulary categorical features [40], which are unique to recommender systems, also spur a series of distinctive SSR methods that can enrich the SSL family.”
• Recommender Systems with Generative Retrieval (2023, background): “As such, it is common practice to maintain dedicated vocabulary for more popular items and randomly hash remaining items in a fixed set of buckets[16].”
• RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, methodology): “Second, hashing permits the handling of unseen inputs at runtime [1, 22].”