DeepRecSys: A System for Optimizing End-To-End At-Scale Neural Recommendation Inference

By Udit Gupta, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S. Lee, David M. Brooks, Carole-Jean Wu. 2020

211 citations recorded by the source corpus.

Neural personalized recommendation is the cornerstone of a wide collection of cloud services and products, constituting significant compute demand of cloud infrastructure. Thus, improving the execution efficiency of recommendation directly translates into infrastructure capacity saving. In this paper, we propose DeepRecSched, a recommendation inference scheduler that maximizes latency-bounded throughput by taking into account characteristics of inference query size and arrival patterns, model architectures, and underlying hardware systems. By carefully optimizing task versus data-level parallelism, DeepRecSched improves system throughput on server class CPUs by $2 \times$ across eight industry-representative models. Next, we deploy and evaluate this optimization in an at-scale production datacenter which reduces end-to-end tail latency across a wide variety of recommendation models by 30%. Finally, DeepRecSched demonstrates the role and impact of specialized AI hardware in optimizing system level performance (QPS) and power efficiency (QPS/watt) of recommendation inference.In order to enable the design space exploration of customized recommendation systems shown in this paper, we design and validate an end-to-end modeling infrastructure, DeepRecInfra. DeepRecInfra enables studies over a variety of recommendation use cases, taking into account at-scale effects, such as query arrival patterns and recommendation query sizes, observed from a production datacenter, as well as industry-representative models and tail latency targets.

Subfields: Artificial Intelligence, Computer Vision and Pattern Recognition, Electrical and Electronic Engineering, Recommender Systems, ML Systems & Hardware

How researchers see this paper

The researchers whose trusted networks rank it highest, strongest advocate first. Every researcher gets a different order.

In Artificial Intelligence

In Computer Vision and Pattern Recognition

In Electrical and Electronic Engineering

In Recommender Systems

In ML Systems & Hardware

Citations in the corpus

Splitwise: Efficient Generative LLM Inference Using Phase Splitting (2023, background): “Prior work exploits such heterogeneity to selectively schedule requests between CPUs and accelerators [38], [52], colocate models with complementary memory usage [30], and partition compute/memory on heterogeneous hardware resources [45], [48].”
TT-Rec: Tensor Train Compression for Deep Learning Recommendation Models (2021, methodology): “To understand Rec-TT training performance for other categories of DLRMs that are embedding-dominated (Gupta et al., 2019; Gupta et al., 2020; Ke et al., 2020), we develop a suite of microbenchmarks to synthetically generate embedding vectors with a polling factor (P) of 10 and 100.”
RecSSD: near data processing for solid state drive based recommendation inference (2021, methodology): “We refer the reader to [19] for detailed operator breakdowns and benchmark model characterizations.”
RecShard: statistical feature-based memory optimization for industry-scale neural recommendation (2022, background): “DLRMs DLRMs exhibit distinct systems implications compared to more traditional neural network architectures [10, 16, 23, 24, 38].”
Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scale (2021, methodology): “Recent works [23, 24, 29, 48] presented in-depth system performance characterization studies and shared the architectural implications for additional optimization opportunities.”
Planaria: Dynamic Architecture Fission for Spatial Multi-Tenant Acceleration of Deep Neural Networks (2020, background): “…Assistants [15, 16], Smart Speakers [17], and enterprise applications [18–20], etc. Cloud-backed inference currently dominates the market [21–24] and is enabled by various forms of custom accelerators, such as Google’s TPU [1], NVIDIA T4 [2], Microsoft Brainwave [3], and Facebook’s DeepRecSys [25].”
CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery (2020, background): “Across clusters ser-vicing machine learning workloads, about 50% of training and 80% of inference cycles are dedicated to recommendation models at Facebook (Gupta et al., 2020).”
Understanding Training Efficiency of Deep Learning Recommendation Models at Scale (2020, background): “In addition, recent studies have started analyzing the system- and architecturelevel implications of neural recommendation inference [20], [21], [24].”
Understanding and Co-designing the Data Ingestion Pipeline for Industry-Scale RecSys Training (2021, background): “DeepRecSys [19] optimized scheduling of inference requests across CPUs and GPUs.”
Domain-specific Genetic Algorithm for Multi-tenant DNNAccelerator Scheduling (2021, methodology): “Lang ALBERT [59], Bart [62], BERT [24], CamemBERT [68], CTRL [52], DistilBERT [91], ELECTRA [18], FlauBERT [60], GPT2 [82], GPT [81], LongFormer [6], MarianNMT [49], MobileBERT [103], Reformer [53], RetriBERT [44], RoBERTa [64], T5 [83], TransformerXL [22], XLM [58], XLM-RoBERTa [19], XLNet [119] Re”
Understanding Capacity-Driven Scale-Out Neural Recommendation Inference (2020, background): “Today, recommendation inference is performed on the CPU, compared to the heterogeneous systems popular with other deep learning workloads [1], [2].”