Deep Learning Recommendation Model for Personalization and Recommendation Systems

By Maxim Naumov, Dheevatsa Mudigere, Hao-Jun Michael Shi, Jianyu Huang, Narayanan Sundaraman, Jongsoo Park, Xiaodong Wang, Udit Gupta, Carole-Jean Wu, Alisson G. Azzolini and 14 more. 2019

944 citations recorded by the source corpus.

With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks. These networks differ significantly from other deep learning networks due to their need to handle categorical features and are not well studied or understood. In this paper, we develop a state-of-the-art deep learning recommendation model (DLRM) and provide its implementation in both PyTorch and Caffe2 frameworks. In addition, we design a specialized parallelization scheme utilizing model parallelism on the embedding tables to mitigate memory constraints while exploiting data parallelism to scale-out compute from the fully-connected layers. We compare DLRM against existing recommendation models and characterize its performance on the Big Basin AI platform, demonstrating its usefulness as a benchmark for future algorithmic experimentation and system co-design.

Subfields: Artificial Intelligence, Information Systems, Evaluation & Benchmarks

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Information Systems

In Evaluation & Benchmarks

Citations in the corpus

Sustainable AI: Environmental Implications, Challenges and Opportunities (2021, background): “RM1 – RM5 represent five unique deep learning recommendation and ranking models for various Facebook products [26], [27].”
A Survey of Quantization Methods for Efficient Neural Network Inference (2021, neutral): “In fact, fake quantization methods can be beneﬁcial for problems that are bandwidth-bound rather than compute-bound, such as in recommendation systems [180].”
MLPerf Training Benchmark (2019, background): “Recommendation systems are a key commercial workload for internet companies (Naumov et al., 2019; Zhou et al., 2018; Cheng et al., 2016).”
AutoTM: Automatic Tensor Movement in Heterogeneous Memory Systems using Integer Linear Programming (2020, background): “Deep Neural Networks (DNNs) have been dramatically successful over the past decade across many domains including computer vision [29], machine translation and language modeling [40], recommendation systems [30], speech [46] and image synthesis [47], and real-time strategy game control [43].”
The Bouncer Problem: Challenges to Remote Explainability (2019, background): “ility in general. 1 Introduction Modern decision-making driven by black-box systems now impacts a signicant share of our lives [14, 38]. These systems build on user data, and range from recommenders [29] (e.g., for personalized ranking of information on websites) to predictive algorithms (e.g., cred”
RecNMP: Accelerating Personalized Recommendation with Near-Memory Processing (2019, methodology): “As a case study, we conduct a thorough characterization of the recentlyreleased Deep Learning Recommendation Model (DLRM) benchmark [49].”
LaMP: When Large Language Models Meet Personalization (2023, background): “Personalization has been widely studied by various communities, including the information retrieval (IR) and human-computer interaction (HCI) communities, often with applications to search engines and recommender systems (Fowler et al., 2015; Xue et al., 2009; Naumov et al., 2019).”
DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference (2020, background): “For example, recent characterization of Facebook’s DLRM implementation demonstrates that DNNs for recommendation have unique compute and memory characteristics [3], [10].”
Towards Automated Neural Interaction Discovery for Click-Through Rate Prediction (2020, methodology): “tures are concatenated into a vector, and its sparse features are embedded into low-dimensional vectors based on the look-up-table operation following similar preprocessing done in various CTR models [21, 30]. Blocks are wired together to form a DAG, and each block could take both raw input features”
Disentangling User Interest and Conformity for Recommendation with Causal Embedding (2021, neutral): “preference from the observational interaction data [36].”
Software-hardware co-design for fast and scalable training of deep learning recommendation models (2021, methodology): “This hybrid parallelism requires AlltoAll communications for the embedding lookup results [42, 43], as well as embedding table input redistribution if the inputs are streamed from database in batches, which is often the case.”
DHEN: A Deep and Hierarchical Ensemble Network for Large-Scale Click-Through Rate Prediction (2022, methodology): “In CTR prediction tasks, the feature inputs usually contain discrete categorical terms (sparse features) and numerical values (dense features) [8, 11, 28, 35].”
Large Language Models for Education: A survey and outlook (2024, neutral): “For example, Leiker et al. [19] present an investigation into the use of LLMs in asynchronous course creation, particularly within the context of adult learning, training, and upskilling.”
Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models (2021, background): “It is not surprising that the community is racing to break the record of the largest parameter counts [Radford et al., 2019, Brown et al., 2020, Dosovitskiy et al., 2020, Tolstikhin et al., 2021, Zhang et al., 2021, Naumov et al., 2019, Jumper et al., 2021].”
DCN V2: Improved Deep & Cross Network and Practical Lessons for Web-scale Learning to Rank Systems (2020, background): “roduct layer as the pairwise interaction layers. One downside of OPNN lies in its high computational cost. Neural FM (NFM) [16] extends FM by replacing the inner-product with a Hadamard product; DLRM [34] follows FM to compute the feature crosses through inner products; These models can only create ”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, methodology): “We evaluate the performance of the different sharding strategies on a system running a modified version of open-source DLRM [7, 33].”
Mixed Dimension Embeddings with Application to Memory-Efficient Recommendation Systems (2019, methodology): “In our experiments we use state-of-the-art Facebook’s DLRM [36] and the Criteo Kaggle dataset [29].”