Neural Input Search for Large Scale Recommendation Models

By Manas R. Joglekar, Cong Li, Jay K. Adams, Pranav Khaitan, Quoc V. Le. 2019

122 citations recorded by the source corpus.

Recommendation problems with large numbers of discrete items, such as products, webpages, or videos, are ubiquitous in the technology industry. Deep neural networks are being increasingly used for these recommendation problems. These models use embeddings to represent discrete items as continuous vectors, and the vocabulary sizes and embedding dimensions, despite their heavy influence on the model's accuracy, are often manually selected in a heuristical manner. We present Neural Input Search (NIS), a technique for learning the optimal vocabulary sizes and embedding dimensions for categorical features. The goal is to maximize prediction accuracy subject to a constraint on the total memory used by all embeddings. Moreover, we argue that the traditional Single-size Embedding (SE), which uses the same embedding dimension for all values of a feature, suffers from inefficient usage of model capacity and training data. We propose a novel type of embedding, namely Multi-size Embedding (ME), which allows the embedding dimension to vary for different values of the feature. During training we use reinforcement learning to find the optimal vocabulary size for each feature and embedding dimension for each value of the feature. Experimentation on two public recommendation datasets shows that NIS can find significantly better models with much fewer embedding parameters. We also deployed NIS in production to a real world large scale App ranking model in our company's App store, Google Play, resulting in +1.02% App Install with 30% smaller model size.

Subfields: Artificial Intelligence, Information Systems, Management Science and Operations Research, Recommender Systems, Neural Architecture Search & AutoML

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Management Science and Operations Research

In Recommender Systems

In Neural Architecture Search & AutoML

Zhenguo Li

Citations in the corpus

Differentiable Neural Input Search for Recommender Systems (2020, methodology): “Feature blocking has been a novel ingredient used in the existing neural input search methods [15, 20] to facilitate the reduction of search space.”
AutoEmb: Automated Embedding Dimensionality Search in Streaming Recommendations (2020, methodology): “It is worth to note that, Neural Input Searchmodel [12] andMixed Dimension Embeddingmodel [7] cannot be applied in the streaming recommendation setting, because they assume that the popularities of users/items are pre-known and fixed, and then assign highlypopular users/items with large embedding di”
DARTS+: Improved Differentiable Architecture Search with Early Stopping (2019, methodology): “The neural architectures searched by NAS have achieved the state-of-the-art results over handcrafted neural architectures in various tasks, including object classiﬁcation [4, 20, 21, 25, 26, 40], object detection [7, 36], semantic segmentation [19], recommender systems [14], etc.”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, background): “As the size of this categorical feature space can be arbitrarily large, it is natural to ask if a subset of values appear more often than others, and in fact they do [8, 19, 25, 42].”
Learning to Embed Categorical Features without Embedding Tables for Recommendation (2020, background): “There are various work on improving the performance or efficiency of embedding learning, via dimensionality search [17], factorization [20], pruning [22], etc.”