Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems

By Antonio A. Ginart, Maxim Naumov, Dheevatsa Mudigere, Jiyan Yang, James Y. Zou. 2019

114 citations recorded by the source corpus.

Embedding representations power machine intelligence in many applications, including recommendation systems, but they are space intensive - potentially occupying hundreds of gigabytes in large-scale settings. To help manage this outsized memory consumption, we explore mixed dimension embeddings, an embedding layer architecture in which a particular embedding vector's dimension scales with its query frequency. Through theoretical analysis and systematic experiments, we demonstrate that using mixed dimensions can drastically reduce the memory usage, while maintaining and even improving the ML performance. Empirically, we show that the proposed mixed dimension layers improve accuracy by 0.1 % using half as many parameters or maintain it using 16 x fewer parameters for click-through rate prediction on the Criteo Kaggle dataset. They also train over 2x faster on a GPU. A full version of this paper is accessible at: https://arxiv.org/abs/1909.11810

Subfields: Artificial Intelligence, Computer Networks and Communications, Information Systems

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Networks and Communications

In Information Systems

Citations in the corpus

AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, background): “It is worth to note that, Neural Input Searchmodel [12] andMixed Dimension Embeddingmodel [7] cannot be applied in the streaming recommendation setting, because they assume that the popularities of users/items are pre-known and fixed, and then assign highlypopular users/items with large embedding di”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, methodology): “Various methods can be used to limit the size of large embedding tables [17], and hashing is a common method for this.”
DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference (2020, unknown): “[71], [72] compresses embedding tables in recommendation models while maintaining the model accuracy.”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, background): “Power-law distributions are a well-known phenomenon of features related to recommender systems [2, 8, 22, 42, 45].”